May 11, 2007 at 1:07 am
I would like your perspective on a (re)design for data aggregation.
This is the general scoop:
- We have many "data providers" grouped in categories.
- Providers in the same category sent data with the same structure.
- Providers can be added/removed dynamically.
- The data provided is linked to other objects in our database.
In the current system, each instance of a data providers writes data
in a separate table. Since we have many providers, we end up with tons
of tables whose names are not known a priori. As a result, we have to
use name conventions for tables, which is not a clean approach.
I considered having only one table for all providers and have a typed
xml column for the actual data. That column is associated with a
schema collection that contains the schemas for all metadata
providers.
scenario. However, I have some concerns:
1. Bulk: Since this table will be very large (possibly millions of
rows).
2. Load: I need to search, add, update, modify entries on this table
from multiple providers. Therefore, there is an issue of contention.
3. Performance: XML is usually very slow.
I would appreciatte your comments in this matter.
What do you think about this approch?
How you compare both designs?
Are there any provisions or techniques to mitigate the concerns with
the simgle xml approach?
Kind regards
CD
May 11, 2007 at 8:08 am
Viewing 2 posts - 1 through 1 (of 1 total)
You must be logged in to reply to this topic. Login to reply