Design the tables to consolidate the kafka messages

• ankit.satyam

  SSC Rookie

  Points: 30

  More actions

  August 2, 2018 at 1:17 pm

  #367012

  Hi All,

  I have below scenario:

  • 2 DBs: DB1 and DB2 which will later replace DB1. Both needs to be in sync. DB1 is de-normalized and 1 table in DB1 might have multiple tables in DB2. When we are loading data from DB2 to DB1 then we need to consolidate the messages to populate a table in DB1 in one shot.

    Need to design a configuration table having below information
    Target_System (DB1 but later can be DB2 as well)
    Target_Table
    Source_table (this will be 1-many with Target Table)
    Mandatory Source Columns/Optional Source columns

    For the entries in this table the messages will be consolidated. To consolidate need to store the messages in landing DB and need to have an indicator set when all the mandatory columns have arrived, combine the message and pass to Target.

    The messages will be in Kafka and Landing DB and configuration table needed in SQL server.

    Thanks.

• Phil Parkin

  SSC Guru

  Points: 246965

  More actions

  August 3, 2018 at 12:16 am

  #2000117

  You have provided a set of requirements, but not asked any questions. Do you need help with something?

  The absence of evidence is not evidence of absence
  - Martin Rees
  The absence of consumable DDL, sample data and desired results is, however, evidence of the absence of my response
  - Phil Parkin

• sgmunson

  SSC Guru

  Points: 110640

  More actions

  August 6, 2018 at 6:55 am

  #2000406

  ankit.satyam - Thursday, August 2, 2018 1:17 PM

  Hi All,

  I have below scenario:

  • 2 DBs: DB1 and DB2 which will later replace DB1. Both needs to be in sync. DB1 is de-normalized and 1 table in DB1 might have multiple tables in DB2. When we are loading data from DB2 to DB1 then we need to consolidate the messages to populate a table in DB1 in one shot.

    Need to design a configuration table having below information
    Target_System (DB1 but later can be DB2 as well)
    Target_Table
    Source_table (this will be 1-many with Target Table)
    Mandatory Source Columns/Optional Source columns

    For the entries in this table the messages will be consolidated. To consolidate need to store the messages in landing DB and need to have an indicator set when all the mandatory columns have arrived, combine the message and pass to Target.

    The messages will be in Kafka and Landing DB and configuration table needed in SQL server.

    Thanks.

  While you provide a number of "specs", you haven't asked any questions, and even your specifications are somewhat vague.   Even if we had a specific question, I'm doubtful an answer would be forthcoming without a fair amount of additional detail.   The purpose of these tables and their data would be of considerable help in understanding what you want to accomplish.   Of course, we would also need to know where your grief is coming from that caused you to post here.

  Steve (aka sgmunson) 🙂 🙂 🙂
  Rent Servers for Income (picks and shovels strategy)