Data migrations are characterized by other factors than classical data extraction and processing (ETL processing). Although data was loaded and transformed from data sources similarly to data warehouse projects, data migration posed special challenges for the team. The following example of an extensive migration demonstrates special features that contribute significantly to the success of the project.
- Successful migration of 99.99% of all data and thus the lowest rework rate compared to other country-specific data migrations in the Group
- Final run time of the migration process: 4 hours - reduction of the originally process by a factor of 6
- Integration of more than 20 complex data sources (various billing, order management and SAP systems)
- Consolidation of 1.2 million heterogeneous customer data into one operative CRM
Incorrect data = Incorrect core business?
Our client in the telecommunication sector aimed to standardize the existing system landscape. In the future, various products such as telephony, Internet and television will be managed via one single system. Different systems (order management, billing, logistics, address management) had to be transferred into a uniform solution. The plan was to convert the existing system world to the new operative CRM with interfaces to the network infrastructure.If data were incorrectly migrated, this would have lea to complex problems with possible medial effects: incorrect shutdown of active customers, claims for damages, incorrect invoice printing - to name only a few examples.
The Ventum migration approach - a combination of classic and agile procedures.
In the context of a migration project, numerous questions had to be clarified in advance prior to the start of a development:
- Which data had to be transferred or could be used by the target system?
- If only parts were taken over, which parts?
- What criteria was used to determine these subsets?
- Was information filtered specifically?
- How was data from multiple source systems synchronized and consolidated?
- How was the old data cleaned up?
- How were existing orders migrated?
In addition, compliance regulations and audit compliance had to be observed; complete traceability of the migration was indispensable (reconciliation). Various tools were used in the project, notably SAP Data Services and OpenTalend, as well as Oracle PL/SQL packages for customer data consolidation.
During analysis and conception phase, a set of decisions were made concerning the migration path, data dependencies with regard to the target system, the data cleansing approach and many other aspects. Depending on previous knowledge of the existing systems, familiarization with the data models was indispensable.
In addition, measurements of the expected workload and running times must be carried out. A realistic time window had to be defined for the final migration and integrated into the overall project plan.
If customer data is was consolidated from different systems, complex business rules had to be defined, e.g. to correct different spellings in name fields and to transfer only the most current and best data.
The aim of the project was to achieve the highest possible degree of automation to build up the migration in a repeatable and robust manner.
The advantages of an agile approach became obvious during development. The basic object was continually extended to the final object, a data package to immigrate data into the target system. This procedure allowed regression tests and qualitative statements for the upcoming cut-over weekend.
The chosen approach started with customer data, followed by additional address and device data and thus further relevant data sources. Finally, pending orders (open work orders) were considered.
After each test migration, the results were evaluated to determine the migration rate. The goal was to reduce the manual rework rate by the call center to a minimum.
Furthermore, data quality problems were transferred to the relevant departments for direct data cleansing and sustainability was measured by further test migrations.
The main factors observed in the cut-over were those that affected the runtime of the migration
Were the source systems available with sufficient performance?
Were all relevant contact persons within reach (war-rooms)?
Were all users removed from the source systems in order to guarantee a uniform database?
During the execution, the performance of individual migration sections was measured and compared with target values from the test migrations in order to maintain the critical time window of the overall project.
After the go-live phase, the migration team received questions from the departments and call centers on individual data records. Fast information was necessary, since the inquiries were mainly about customer contact data.
Within the scope of the project, a reporting system was developed for this purpose in order to provide departments information on data origin and transformation (self-service). Complex analyses were performed by the migration team.
99.97% migration rate - minimal manual post-processing and fast focus on core business
Due to the high migration rate, post-processing had to be reduced to a minimum. The focus of the customer employees was on the core business, sales and customer service from day 1 after the changeover.
Ventum brought expert knowledge of a variety of enterprise applications and host development to the project so that the analysis phase could be shortened. By focusing on data quality and the chosen step-by-step migration approach, this subproject was able to make a significant contribution to the success of the project.
- Fully automated data migration of 99.97% of customer accounts within the scheduled time window
- Amortization of the entire project within 2 years simply by saving postage costs
- Improvement of service quality through comprehensive data cleansing
- Release of documentation by internal audit