Data migrations are characterized by other factors than classical data extraction and processing (ETL processing). Although data are loaded and transformed from data sources similarly to data warehouse projects, data migration poses special challenges for the team. The following example of an extensive migration demonstrates special features that contribute significantly to the success of the project.
- Successful migration of 99.99% of all data and thus the lowest rework rate compared to other country-specific data migrations in the Group
- Final run time of the migration process: 4 hours - reduction of the originally process by a factor of 6
- Integration of more than 20 complex data sources (various billing, order management and SAP systems)
- Consolidation of 1.2 million heterogeneous customer data into one operative CRM
Incorrect data = Incorrect core business?
Our client in the telecommunication sector aims to standardize the existing system landscape. In the future, various products such as telephony, Internet and television will be managed via one single system. Different systems (order management, billing, logistics, address management) have to be transferred into a uniform solution. The plan is to convert the existing system world to the new operative CRM with interfaces to the network infrastructure.
If data were incorrectly migrated , this would lead to complex problems with possible medial effects: incorrect shutdown of active customers, claims for damages, incorrect invoice printing - to name only a few examples..
The Ventum migration approach - a combination of classic and agile procedures.
In the context of a migration project, numerous questions have to be clarified in advance prior to the start of a development:
- Which data should be transferred or can be used by the target system?
- If only parts are taken over, which parts?
- What criteria are used to determine these subsets?
- Is information filtered specifically?
- How are data from multiple source systems synchronized and consolidated?
- How is the old data cleaned up?
- How are existing orders migrated?
In addition, compliance regulations and audit compliance must be observed; complete traceability of the migration is indispensable (reconciliation). Various tools were used in the project, in particular SAP Data Services and OpenTalend as well as Oracle PL/SQL packages for customer data consolidation.
During analysis and conception phase, a set of decisions has to be made concerning the migration path, data dependencies with regard to the target system, the data cleansing approach and many other aspects. Depending on previous knowledge of the existing systems, familiarization with the data models might be indispensable.
In addition, measurements of the expected workload and running times must be carried out. A realistic time window must be defined for the final migration and integrated into the overall project plan.
If customer data is to be consolidated from different systems, complex business rules have to be defined, e.g. to correct different spellings in name fields and to transfer only the most current and best data.
The aim of the project was to achieve the highest possible degree of automation in order to build up the migration in a repeatable and robust manner.
The advantages of an agile approach become obvious during development. The basic object is continually extended to the final object, a data package to immigrate data into the target system. This procedure allows regression tests and qualitative statements for the upcoming cut-over weekend.
The chosen approach started with customer data, followed by additional address and device data and thus further relevant data sources. Finally, pending orders (open work orders) were taken into account.
After each test migration, the results were evaluated to determine the migration rate. The goal was to reduce the manual rework rate by the call center to a minimum.
Furthermore, data quality problems were transferred to the relevant departments for direct data cleansing and sustainability was measured by further test migrations.
The main factors to be observed in the cut-over are those that affect the runtime of the migration
Are the source systems available with sufficient performance?
Are all relevant contact persons within reach (war-rooms)?
Are all users removed from the source systems in order to guarantee a uniform database?
During the execution, the performance of individual migration sections is measured and compared with target values from the test migrations in order to maintain the critical time window of the overall project.
After the go-live phase, the migration team will receive questions from the departments and call centers on individual data records. Fast information is necessary, since the inquiries are mainly about customer contact data.
Within the scope of the project, a reporting system was developed for this purpose in order to provide departments information on data origin and transformation (self-service). Complex analyses were performed by the migration team.
99.97% migration rate - minimal manual post-processing and fast focus on core business
Due to the high migration rate, post-processing could be reduced to a minimum. The focus of the customer employees was on the core business, sales and customer service from day 1 after the changeover.
Ventum brought expert knowledge of a variety of enterprise applications and host development to the project so that the analysis phase could be shortened. By focusing on data quality and the chosen step-by-step migration approach, this subproject was able to make a significant contribution to the success of the project.
- Fully automated data migration of 99.97% of customer accounts within the scheduled time window
- Amortization of the entire project within 2 years simply by saving postage costs
- Improvement of service quality through comprehensive data cleansing
- Release of documentation by internal audit