Post reply

removing duplicates

  • November 24, 2008 at 3:36 pm

    #128846

    Hi All,

    I am having a requirement to remove the duplicates that are coming in my flow after the union all transformatin..

    I am having 12 sources and am doing union all in 2 columns finally...

    but they are fetching duplicates and even though i am doing lookup on the key that i am fetching as it will not found there in the lookup it is loading n times in the destination..

    i used SORT transformation and removed duplicates...

    but i dont like to use this transformation...

    I there any other way to remove duplicate values in the flow..

    Thanks for your help

    [font="Comic Sans MS"]
    ---------------------------------------------------

    Thanks [/font]

  • November 25, 2008 at 7:15 am

    #903112

    no option to remove duplicates?

    [font="Comic Sans MS"]
    ---------------------------------------------------

    Thanks [/font]

  • November 25, 2008 at 7:22 am

    #903119

    Can you explain why you don't want to use the Sort component?

    You can handle the removal of duplicates by inserting your data first in a staging table and then handling it there.

    ---
    SSIS Tasks Components Scripts Services | http://www.cozyroc.com/

  • November 25, 2008 at 8:31 am

    #903191

    Try using UNION on its own rather than UNION ALL, which ensures that duplicates are retained in the output.

    “Write the query the simplest way. If through testing it becomes clear that the performance is inadequate, consider alternative query forms.” - Gail Shaw

    For fast, accurate and documented assistance in answering your questions, please read this article.
    Understanding and using APPLY, (I) and (II) Paul White
    Hidden RBAR: Triangular Joins / The "Numbers" or "Tally" Table: What it is and how it replaces a loop Jeff Moden

Viewing 4 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic. Login to reply