ETL is used by the Data Engineers, ETL or ELT Developers and involves the learning process for various tools and the architecture’s used. With the emergence of the big data and different formats , the ETL process used for data migration becomes sometimes complex. The purpose of this web site is to discuss various scenarios based on the Architecture , Tools used , Performance, Level of effort involved in development and

Data Migration Scenarios

1. Migration of Data for Data Warehouses
2. Migration of Data for Application Upgrades
3. Migration of Data for New Applications
4. Migration of Data for Data Analytics

The above scenarios , just came in my mind , there can be many other reasons for the organization to migrate and transform the data.

Misconception’s

First misconception and the most common is to determine the scope of the migration project. The people involved in determining the scope of the data migration project are most of the time unaware of the importance of the data migration as they are mostly business users and project managers who have very little experience as developers. The front end development is reviewed with the business users and they provide lots of feedback and their is so much back and forth , that happens during the application development meetings , that the end user is mostly aware of the product that they are getting.

Data Migration is a backend process and the only motivation for the developer , that is working on the data migration is himself. Once the data is migrated and all the validations are done on the data and there are very few errors , the developer who has developed the migration scripts and the transformations used , needs to give a pat on his or her back , since there is very less praise seen from the business users as they are not aware of the effort involved in data cleansing and transformation of data. A Business users just thinks that this is just the copying of the data from once database to another (Ctrl+ C and Ctrl +V).

Tools and Languages

There are various tools available in the market by Microsoft, Informatica, Data Bricks, Oracle are the few that come in my mind and are mostly used across different industries.