Multi-file U-SQL query doesn't return correct results

Question

Multi-file U-SQL query doesn't return correct results

Michelle-138172

Hall of Fame

Points: 3234
More actions
March 19, 2018 at 9:56 am

#333951

Looks pretty quiet here, but still hope to get some answers.
I'm a new user to U-SQL. I have a query that queries files in 2 folders for aggregation, but the result comes back with only partial counts for the 2nd folder. If I filter the result with only the 2nd folder, the result looks fine. Wondering why this is happening. How can I get the full result back for the multi-files?

Viewing 9 posts - 1 through 8 (of 8 total)

You must be logged in to reply to this topic. Login to reply

Lynn Pettis SSC Guru Points: 442467 More actions · Answer 1

Michelle-138172 - Monday, March 19, 2018 9:56 AM
Looks pretty quiet here, but still hope to get some answers.
I'm a new user to U-SQL. I have a query that queries 2 files for aggregation, but the result comes back with only partial counts for the 2nd file. If I filter the result with only the 2nd file, the result looks fine. Wondering why this is happening. How can I get the full result back for the multi-files?

Unfortunately, no one can see what you see. Without more details anything we say will be just wild shots in the dark.

Michelle-138172 Hall of Fame Points: 3234 More actions · Answer 2

Thanks for the reply, Lynn. I actually meant 2 folders, not 2 files. It's almost like it's not reading all the files in the 2nd folder.

It's a simple count aggregation by date and another column. It works fine if I do a single date, but once I comment out the date filter, the result is skewed. Very puzzling.

Lynn Pettis SSC Guru Points: 442467 More actions · Answer 3

Michelle-138172 - Monday, March 19, 2018 10:35 AM
Thanks for the reply, Lynn. I actually meant 2 folders, not 2 files. It's almost like it's not reading all the files in the 2nd folder.
It's a simple count aggregation by date and another column. It works fine if I do a single date, but once I comment out the date filter, the result is skewed. Very puzzling.

Hopefully someone else will be able to help, but with nothing to work with here I wouldn't even know where to start.

Michelle-138172 Hall of Fame Points: 3234 More actions · Answer 4

I added one more day's worth of data for testing. It looks like the output file is really just cutting off at a certain row number. In this case, it always ends at row 102. Is there any default row limit setting?

Lynn Pettis SSC Guru Points: 442467 More actions · Answer 5

If you really want help you will need to post details about what you are trying to accomplish. We aren't mind readers and we can't see what you are doing unless you show us.

Michelle-138172 Hall of Fame Points: 3234 More actions · Answer 6

OK, I figured this out. It was a rookie mistake. I was in the preview mode. When I fully downloaded the file, I got everything.

Lynn Pettis SSC Guru Points: 442467 More actions · Answer 7

Glad you figured it out. Not something I would have even thought to ask about.

Sue_H SSC Guru Points: 90860 More actions · Answer 8

Michelle-138172 - Monday, March 19, 2018 1:50 PM
I added one more day's worth of data for testing. It looks like the output file is really just cutting off at a certain row number. In this case, it always ends at row 102. Is there any default row limit setting?

But no one can see your query so all anyone can do is guess. You'll get one post "are you using sample", another one "are you using offset", another one "are you using fetch". And it goes on and on like that and eventually people stop trying to guess. The more information people provide when posting, the more involved others others seem to be. It's the best way to get help when posting.

Sue