Please start any new threads on our new
site at https://forums.sqlteam.com. We've got lots of great SQL Server
experts to answer whatever question you can come up with.
| Author |
Topic |
|
dewacorp.alliances
452 Posts |
 Posted - 2008-11-20 : 20:18:34
|
| Hi thereWe have flat file and as apart of ETL process, we bring this flat into a mapped table (called tblSourceFlatFile for instance and it's mapped out as it is with few additional column). My question is: Before doing any tranfrom/messaging from this table into a destination table, is it worthwile to do Indexing and Update Statistic or not on this table (tblSourceFlatFile)? The new data is changing all the time and eahc load is an average of 2 million records and we need to load/tranform this data everytime.I'm appreciated your comment/thought.Thanks |
|
|
nr
SQLTeam MVY
12543 Posts |
Posted - 2008-11-20 : 20:34:40
|
| It depends on what you are doing to the data.Could you do the work in the SSIS package in the pipeline rather than load it into the table? Or maybe do some more work to make it easier later. Often lookups aren't good in SSIS (depends on what you are doing).If you are using the data from this table to use 9indexes on other tables then indexes might not help but if you are doing aggregations or such they might well.Depends really on thesituation - without more info it's not possible to say.==========================================Cursors are useful if you don't know sql.DTS can be used in a similar way.Beer is not cold and it isn't fizzy. |
 |
|
|
dewacorp.alliances
452 Posts |
Posted - 2008-11-20 : 21:55:04
|
quote:
Originally posted by nr
It depends on what you are doing to the data. Could you do the work in the SSIS package in the pipeline rather than load it into the table? Or maybe do some more work to make it easier later. Often lookups aren't good in SSIS (depends on what you are doing).If you are using the data from this table to use 9indexes on other tables then indexes might not help but if you are doing aggregations or such they might well.
I have to load this data cause we need to capture this flat file into a table as apart of the requirement. Then data within this table will be transformed into a destination table with some filtering and calculation etc etc (for instance we select column1 with code='XXX' and do the manipulation/sum/avg/whatever) and then transfor to a 'normalised' table.Hope this will help. |
|
|
|
nr
SQLTeam MVY
12543 Posts |
Posted - 2008-11-21 : 02:52:27
|
| well maybe clustered index on code - or maybe non-clustered index which is covering or combined with clustered index on a unique id==========================================Cursors are useful if you don't know sql.DTS can be used in a similar way.Beer is not cold and it isn't fizzy. |
|
|
|
|
|
|
Indexing and Update Statistic on Source Data