Analisis Dynamic ETL Incremental Load untuk Data Integration Datawarehouse
DOI:
https://doi.org/10.32627/internal.v4i2.260Keywords:
Data Integration, ETL, SSIS, data warehouse, multiple sourceAbstract
Data integration is a combination of techniques and businesses that are used to collect data from different sources into useful and valuable information ETL process that includes extracting data from various data sources, transforming data to form and calculate data and load data on target storage, to support data warehouse need. Based on organizations and industries that have implemented data warehouse, the problem that generally arises regarding data load is the difficulty in integrating different data sources, how to form data from various data formats into uniform data, how to integrate data delta between data sources and target storage in an incremental load process so that this data synchronization process can be carried out continuously and relatively faster. ETL process requires a platform that can facilitate data integration needs, in order to run this process. SSIS (SQL Server Integration Service) is a Data Integration platform to build an enterprise-level data integration and solutions for data transformation. Integration Service can extract and change data (transform) from various sources such as XML data files, flat files, APIs, and relational data sources, and then load into one or several destination data. According to the problem related to data load, we will examine how the solution model uses SSIS for the ETL process. This paper proposed an ETL Architecture model by completing the ETL process for full & incremental load extraction and the original data layer.
References
A. B. O. J. G. Philip Woodall, "Data Quality Problems in ETL: The State of the Practice in Large Organisations," IOSR Journal of Computer Engineering (IOSR-JCE), vol. 16, no. 1, p. 18, 2016.
D. Laudenschlager, 06 2018. [Online]. Available: https://docs.microsoft.com /en-us/sql/integration-services/sql-server-integration-services?view=sql-server-2017.
R. H. Junya, "Heterogeneous Database Synchronization Mechanism Based on ETL and XML," RISTI, p. 154, 2015.
K. Kakish and T. Kraft, "ETL Evolution for Real-Time Data Warehousing," in Proceedings of the Conference on Information Systems Applied Research, New Orleans Louisiana, 2012.
R. Kimball, L. Reeves, M. Ross, and W. Thornthwaite, The Data Warehouse Lifecycle Toolkit: Export Methods for Designing,Developing and Developing and Deploying Data Warehouses, Indiana: Wiley Publishing Inc, 1998.
B. t. D. Warehouse, Inmon, W. H., Indiana: Wiley Publishing Inc, 1996.
A. Jain, S. Garg, and N. Sharma, "The Management of Conformed ETL Architecture," International Journal of Computer Application, vol. 118, p. 20, 2015.
D. Laudenschlager, G. Milener, D. Mabee, M. B, and Craig Guyer, “SQL Server Integration Services,” 2018. [Online]. Available: https://docs.microsoft.com/en-us/sql/integration- services/sql-server-integration-services?view=sql-server-2017. [Accessed: 09-Dec-2018]
P. Ponniah, FUNDAMENTALS DATA WAREHOUSING FUNDAMENTALS A Comprehensive Guide for, vol. 6. 2001.
P. Bindal and P. Khurana, “ETL Life Cycle,” Citeseer, vol. 6, no. 2, pp. 1787–1791, 2015.
Lv, Junya, and H. Ren, “Heterogeneous database synchronization mechanism based on ETL and XML,” RISTI [Revista Iber. Sist. e Tecnol. Inf., vol. no. 17A, p. 153+., 2016.
M. Thesis and A. Mashkoor, “Investigating Model Driven Architecture,” 2004.
Microsoft Team, “Integration Services Programming Overview.” [Online]. Available: https://docs.microsoft.com/en-us/sql/integration-services/integration-services-programming-overview?view=sql-server-ver15.
N. D. M. Hwy, “DAMA International DAMA International Handbook Revised?: October 2009,” no. October 2009.
K. Krishnan, Data Warehousing in the Age of Big Data. Morgan Kaufmann, 2013.
