Data Integration - Revision history

User at 23:25, 22 March 2024

2024-03-22T23:25:02Z

User at 12:08, 24 February 2024

2024-02-24T12:08:48Z

User at 14:12, 25 July 2023

2023-07-25T14:12:45Z

User at 14:06, 15 March 2023

2023-03-15T14:06:02Z

User at 15:49, 2 February 2023

2023-02-02T15:49:06Z

User at 03:16, 19 January 2023

2023-01-19T03:16:36Z

Show changes

User at 17:45, 11 November 2022

2022-11-11T17:45:38Z

Show changes

User: The LinkTitles extension automatically added links to existing pages (https://github.com/bovender/LinkTitles).

2021-02-06T15:09:39Z

The LinkTitles extension automatically added links to existing pages (https://github.com/bovender/LinkTitles).

Show changes

User: Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. A complete data integration solution delivers trusted data from various sources.

2018-12-19T19:20:37Z

Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. A complete data integration solution delivers trusted data from various sources.

User: Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. A complete data integration solution delivers trusted data from various sources.

2018-12-19T19:01:35Z

Show changes

@@ Line 97: / Line 97: @@
 == See Also ==
 *[[Application Integration]]
 *[[Big Data Integration]]

@@ Line 97: / Line 97: @@
 == See Also ==
 *[[Application Integration]]
 *[[Big Data Integration]]

← Older revision		Revision as of 14:12, 25 July 2023
Line 10:		Line 10:

	== Data Integration Historical Background<ref>[https://en.wikipedia.org/wiki/Data_integration Historical Background of Data Integration]</ref> ==		== Data Integration Historical Background<ref>[https://en.wikipedia.org/wiki/Data_integration Historical Background of Data Integration]</ref> ==
−	Issues with combining heterogeneous data sources, often referred to as information silos, under a single query interface have existed for some time. In the early 1980s, computer scientists began designing systems for the interoperability of heterogeneous databases. The first data integration system driven by structured metadata was designed at the University of Minnesota in 1991, for the Integrated Public Use Microdata Series (IPUMS). IPUMS used a data warehousing approach, which extracts, transforms, and loads data from heterogeneous sources into a single view schema so data from different sources become compatible. By making thousands of population databases interoperable, IPUMS demonstrated the feasibility of large-scale data integration. The data warehouse approach offers a tightly coupled architecture because the data are already physically reconciled in a single queryable repository, so it usually takes a little time to resolve queries. The data warehouse approach is less feasible for datasets that are frequently updated, requiring the extract, transform, load (ETL) process to be continuously re-executed for synchronization. Difficulties also arise in constructing data warehouses when one has only a query interface to summarize data sources and no access to the full data. This problem frequently emerges when integrating several commercial query services like travel or classified advertisement web applications. As of 2009, the trend in data integration favored loosening the coupling between data and providing a unified query interface to access real-time data over a mediated schema, which allows information to be retrieved directly from original databases. This is consistent with the SOA approach popular in that era. This approach relies on mappings between the mediated schema and the schema of original sources and transforms a query into specialized queries to match the schema of the original databases. Such mappings can be specified in 2 ways: as a mapping from entities in the mediated schema to entities in the original sources (the "Global As View" (GAV) approach), or as a mapping from entities in the original sources to the mediated schema (the "Local As View" (LAV) approach). The latter approach requires more sophisticated inferences to resolve a query on the mediated schema but makes it easier to add new data sources to a (stable) mediated schema. As of 2010 some of the work in data integration research concerns the semantic integration problem. This problem addresses not the structuring of the architecture of the integration, but how to resolve semantic conflicts between heterogeneous data sources. For example, if two companies merge their databases, certain concepts and definitions in their respective schemas like "earnings" inevitably have different meanings. In one database it may mean profits in dollars (a floating-point number), while in the other it might represent the number of sales (an integer). A common strategy for the resolution of such problems involves the use of ontologies that explicitly define schema terms and thus help to resolve semantic conflicts. This approach represents ontology-based data integration. On the other hand, the problem of combining research results from different bioinformatics repositories requires bench-marking of the similarities, computed from different data sources, on a single criterion such as positive predictive value. This enables the data sources to be directly comparable and can be integrated even when the natures of experiments are distinct. As of 2011 it was determined that current data modeling methods were imparting data isolation into every data architecture in the form of islands of disparate data and information silos. This data isolation is an unintended artifact of the data modeling methodology that results in the development of disparate data models. Disparate data models, when instantiated as databases, form disparate databases. Enhanced data model methodologies have been developed to eliminate the data isolation artifact and to promote the development of integrated data models. One enhanced data modeling method recasts data models by augmenting them with structural metadata in the form of standardized data entities. As a result of recasting multiple data models, the set of recast data models will now share one or more commonality relationships that relate the structural metadata now common to these data models. Commonality relationships are peer-to-peer types of entity relationships that relate to the standardized data entities of multiple data models. Multiple data models that contain the same standard data entity may participate in the same commonality relationship. When integrated data models are instantiated as databases and are properly populated from a common set of master data, then these databases are integrated. Since 2011, data hub approaches have been of greater interest than fully structured (typically relational) Enterprise Data Warehouses. Since 2013, data lake approaches have risen to the level of Data Hubs. These approaches combine unstructured or varied data into one location but do not necessarily require an (often complex) master relational schema to structure and define all data in the Hub.	+	Issues with combining heterogeneous data sources, often referred to as information silos, under a single query interface have existed for some time. In the early 1980s, computer scientists began designing systems for the interoperability of heterogeneous databases. The first data integration system driven by structured metadata was designed at the University of Minnesota in 1991, for the Integrated Public Use Microdata Series (IPUMS). IPUMS used a data warehousing approach, which extracts, transforms, and loads data from heterogeneous sources into a single view schema so data from different sources become compatible. By making thousands of population databases interoperable, IPUMS demonstrated the feasibility of large-scale data integration. The data warehouse approach offers a tightly coupled architecture because the data are already physically reconciled in a single queryable repository, so it usually takes a little time to resolve queries. The data warehouse approach is less feasible for datasets that are frequently updated, requiring the extract, transform, load (ETL) process to be continuously re-executed for synchronization. Difficulties also arise in constructing data warehouses when one has only a query interface to summarize data sources and no access to the full data. This problem frequently emerges when integrating several commercial query services like travel or classified advertisement web applications. As of 2009, the trend in data integration favored loosening the coupling between data and providing a unified query interface to access real-time data over a mediated schema, which allows information to be retrieved directly from original databases. This is consistent with the SOA approach popular in that era. This approach relies on mappings between the mediated schema and the schema of original sources and transforms a query into specialized queries to match the schema of the original databases. Such mappings can be specified in 2 ways: as a mapping from entities in the mediated schema to entities in the original sources (the "Global As View" (GAV) approach), or as a mapping from entities in the original sources to the mediated schema (the "Local As View" (LAV) approach). The latter approach requires more sophisticated inferences to resolve a query on the mediated schema but makes it easier to add new data sources to a (stable) mediated schema. As of 2010 some of the work in data integration research concerns the semantic integration problem. This problem addresses not the structuring of the architecture of the [[integration]], but how to resolve semantic conflicts between heterogeneous data sources. For example, if two companies merge their databases, certain concepts and definitions in their respective schemas like "earnings" inevitably have different meanings. In one database it may mean profits in dollars (a floating-point number), while in the other it might represent the number of sales (an integer). A common strategy for the resolution of such problems involves the use of ontologies that explicitly define schema terms and thus help to resolve semantic conflicts. This approach represents ontology-based data integration. On the other hand, the problem of combining research results from different bioinformatics repositories requires bench-marking of the similarities, computed from different data sources, on a single criterion such as positive predictive value. This enables the data sources to be directly comparable and can be integrated even when the natures of experiments are distinct. As of 2011 it was determined that current data modeling methods were imparting data isolation into every data architecture in the form of islands of disparate data and information silos. This data isolation is an unintended artifact of the data modeling methodology that results in the development of disparate data models. Disparate data models, when instantiated as databases, form disparate databases. Enhanced data model methodologies have been developed to eliminate the data isolation artifact and to promote the development of integrated data models. One enhanced data modeling method recasts data models by augmenting them with structural metadata in the form of standardized data entities. As a result of recasting multiple data models, the set of recast data models will now share one or more commonality relationships that relate the structural metadata now common to these data models. Commonality relationships are peer-to-peer types of entity relationships that relate to the standardized data entities of multiple data models. Multiple data models that contain the same standard data entity may participate in the same commonality relationship. When integrated data models are instantiated as databases and are properly populated from a common set of master data, then these databases are integrated. Since 2011, data hub approaches have been of greater interest than fully structured (typically relational) Enterprise Data Warehouses. Since 2013, data lake approaches have risen to the level of Data Hubs. These approaches combine unstructured or varied data into one location but do not necessarily require an (often complex) master relational schema to structure and define all data in the Hub.

@@ Line 4: / Line 4: @@
 Data integration primarily supports the analytical processing of large data sets by aligning, combining, and presenting each data set from organizational departments and external remote sources to fulfill integrator objectives. Data integration is generally implemented in data warehouses (DW) through specialized software that hosts large data repositories from internal and external resources. Data is extracted, amalgamated, and presented in a unified form. For example, a user’s complete data set may include extracted and combined data from marketing, sales, and operations, which is combined to form a complete report. An example of data integration in a smaller paradigm is spreadsheet integration in a Microsoft Word document.<ref>[https://www.techopedia.com/definition/28290/data-integration Explaining Data Integration]</ref>
-Data Integration creates a unified view that can help inform critical business processes in the future. In a commercial situation, the overall goal of data integration is to connect and combine business information that, when initially collected, may have been placed into separate silos managed by unique stakeholders. Generally, these silos are overseen by business units that have little regular interaction with each other, causing a lack of shared insight that can result in missed opportunities for the company as a whole. Data integration helps to unite these siloes, empowering the company to take a holistic approach to serve each customer, asset, or initiative. Data integration may take place in a physical or virtual data warehouse, utilizing software that hosts large repositories of data from internal and external resources. One approach is to use an extract, transform and load method (ETL) to extract the necessary data given the integration objective, amalgamate the relevant information, and then present it as a unified set of content for analysis.<ref>[https://www.pitneybowes.com/us/customer-information-management/case-studies/what-is-data-integration.html What is Data Integration?]</ref>
+Data Integration creates a unified view that can help inform critical business processes in the future. In a commercial situation, the overall goal of data integration is to connect and combine business information that, when initially collected, may have been placed into separate silos managed by unique stakeholders. Generally, these silos are overseen by business units that have little regular interaction with each other, causing a lack of shared insight that can result in missed opportunities for the company as a whole. Data integration helps to unite these siloes, empowering the company to take a holistic approach to serve each customer, asset, or initiative. Data integration may take place in a physical or virtual data warehouse, utilizing software that hosts large repositories of data from internal and external resources. One approach is to use an extract, transform, and load method (ETL) to extract the necessary data given the integration objective, amalgamate the relevant information, and then present it as a unified set of content for analysis.<ref>[https://www.pitneybowes.com/us/customer-information-management/case-studies/what-is-data-integration.html What is Data Integration?]</ref>
-Data integration tools have traditionally been delivered via a set of related markets, with vendors in each market offering a specific style of data integration tool. In recent years, most of the activity has been within the ETL tool market. Markets for replication tools, data federation (EII) and other submarkets each included vendors offering tools optimized for a particular style of data integration, and periphery markets (such as data quality tools, adapters, and data modeling tools) also overlapped with the data integration tool space. The result of all this historical fragmentation in the markets is the equally fragmented and complex way in which data integration is accomplished in large enterprises — different teams using different tools, with little consistency, lots of overlap and redundancy, and no common management or leverage of metadata. Technology buyers have been forced to acquire a portfolio of tools from multiple vendors to amass the capabilities necessary to address the full range of their data integration requirements. This situation is now changing, with the separate and distinct data integration tool submarkets converging at the vendor and technology levels. This is being driven by buyer demands as organizations realize they need to think about data integration holistically and have a common set of data integration capabilities they can use across the enterprise. It is also being driven by the actions of vendors, such as those in individual data integration submarkets organically expanding their capabilities into neighboring areas, as well as by acquisition activity that brings vendors from multiple submarkets together. The result is a market for complete data integration tools that address a range of different data integration styles and are based on common design tooling, metadata, and runtime architecture.<ref>[https://www.gartner.com/it-glossary/data-integration-tools/ Understanding Data Integration]</ref>
+Data integration tools have traditionally been delivered via a set of related markets, with vendors in each market offering a specific style of data integration tool. In recent years, most of the activity has been within the ETL tool market. Markets for replication tools, data federation (EII), and other submarkets each included vendors offering tools optimized for a particular style of data integration, and periphery markets (such as data quality tools, adapters, and data modeling tools) also overlapped with the data integration tool space. The result of all this historical fragmentation in the markets is the equally fragmented and complex way in which data integration is accomplished in large enterprises — different teams using different tools, with little consistency, lots of overlap and redundancy, and no common management or leverage of metadata. Technology buyers have been forced to acquire a portfolio of tools from multiple vendors to amass the capabilities necessary to address the full range of their data integration requirements. This situation is now changing, with the separate and distinct data integration tool submarkets converging at the vendor and technology levels. This is being driven by buyer demands as organizations realize they need to think about data integration holistically and have a common set of data integration capabilities they can use across the enterprise. It is also being driven by the actions of vendors, such as those in individual data integration submarkets organically expanding their capabilities into neighboring areas, as well as by acquisition activity that brings vendors from multiple submarkets together. The result is a market for complete data integration tools that address a range of different data integration styles and are based on common design tooling, metadata, and runtime architecture.<ref>[https://www.gartner.com/it-glossary/data-integration-tools/ Understanding Data Integration]</ref>
@@ Line 97: / Line 97: @@
 == See Also ==
 *[[Application Integration]]
 *[[Big Data Integration]]

@@ Line 65: / Line 65: @@
 Data integration seeks to solve many of the following issues that come about when you have disparate information stored in different applications across an organization.
 *Data Silos: A data silo, much like the grain silo it is named after, is a repository of data that is isolated. Generally, in businesses, this means that the information is under the control of a business unit or department and is not available across the organization. This can also occur when an organization has stored information in software that is incompatible. For example, maybe you have some of your marketing data in Salesforce, other data in Marketo, and still more in a database maintained by your Marketing team. But since these systems don't communicate, the information in each application is siloed, so you might be able to draw conclusions from the Salesforce data and the Marketo data, but you won't be able to bring those information sources together to understand the information in totality.
-*Slow Analysis: Business leaders agree that today's decision-making is heavily dependent on good information. Yet, even though they rely upon good data, companies are often frustrated by the amount of time it takes to integrate it. If your data is spread across multiple teams, databases, and applications, it can take a long time to gather and process the data so you can analyze it. And if the process takes long enough, the data will be outdated by the time you have a chance to analyze it. Business decisions need to be made in real-time, and the way to do that is to have a system in place to integrate data before you need it.
+*Slow Analysis: Business leaders agree that today's decision-making is heavily dependent on good information. Yet, even though they rely upon good data, companies are often frustrated by the amount of time it takes to integrate it. If your data is spread across multiple teams, databases, and applications, it can take a long time to gather and process the data so you can analyze it. And if the process takes long enough, the data will be outdated by the time you have a chance to analyze it. Business decisions need to be made in real time, and the way to do that is to have a system in place to integrate data before you need it.
 *Deep Dives: When your data is scattered across different sources and applications, it's difficult to have a complete view of it. For example, maybe you have customer data from different devices and apps. Maybe you have data on purchases from your different storefronts and online sites, but you want to correlate that data with your customer information and you want to enrich it with timestamps and geographical information for a deep analysis of your sales data. If your systems aren't integrated, or if the data isn't compatible, you won't be able to correlate this information without considerable time and effort.
-== Benefits of Data Integration'''<ref>[https://www.talend.com/resources/what-is-data-integration/ Benefits of Data Integration]</ref> ==
+== Benefits of Data Integration<ref>[https://www.talend.com/resources/what-is-data-integration/ Benefits of Data Integration]</ref> ==
 Even if a company is receiving all the data it needs, that data will most often reside in a number of separate data sources. Information from all of those different sources often needs to be pulled together for analysis, and that can be no small task for a professional trying to create a report or make an informed management decision. Without unified data, a single report typically involves logging into multiple accounts, on multiple sites, accessing data within native apps, copying over the data, reformatting, and cleansing, all before analysis can happen. This is central to the importance of data integration, and it brings us to the major benefits of these efforts:
 *Data Integration Improves Collaboration and Unification of Systems: Employees in every department — and sometimes in disparate physical locations — increasingly need access to the company's data for shared and individual projects. IT needs a secure solution for delivering data via self-service access across all lines of business. Additionally, employees in almost every department are generating and improving data that the rest of the business needs. Data integration needs to be collaborative and unified in order to improve collaboration and unification across the organization.
@@ Line 85: / Line 85: @@
 *Implementation Challenges
 **Tool Selection: The tool selection is also very important to the overall success of any data integration project. A feasibility study should be performed to select the right toolset for the job.
-Small enterprises or companies who are just starting their data warehousing initiative are faced with this challenge and sometimes, making that decision isn’t easy considering the number of options available today. The feasibility study helps map out which tools are best suited for the overall data integration objective for the organization. Sometimes, organizations that have built their data warehouse and a lot of ETL processes using a tool that is no more able to scale or the tool is no more relevant. For example, organizations that have invested years of work in OWB are facing that challenge as Oracle decided to do away with OWB to promote a much better tool, ODI. In these cases, even mature organizations need to do a feasibility analysis to estimate what it takes to upgrade the existing Data Integration infrastructure to the new toolset.
+Small enterprises or companies who are just starting their data warehousing initiative are faced with this challenge and sometimes, making that decision isn’t easy considering the number of options available today. The feasibility study helps map out which tools are best suited for the overall data integration objective for the organization. Sometimes, organizations that have built their data warehouse and a lot of ETL processes use a tool that is no more able to scale or the tool is no more relevant. For example, organizations that have invested years of work in OWB are facing that challenge as Oracle decided to do away with OWB to promote a much better tool, ODI. In these cases, even mature organizations need to do a feasibility analysis to estimate what it takes to upgrade the existing Data Integration infrastructure to the new toolset.
@@ Line 97: / Line 97: @@
 == See Also ==
-[[Application Integration]]
+*[[Application Integration]]
@@ Line 104: / Line 106: @@
-===Further Reading===
+== Further Reading ==
 *[http://hosteddocs.toolbox.com/SevenStylesofDataIntegration.pdf Seven Styles of Data Integration]
 *[https://searchdatamanagement.techtarget.com/tutorial/Structuring-data-integration-models-and-data-integration-architecture Structuring data integration models and data integration architecture]

@@ Line 34: / Line 34: @@
 *Data Consolidation: [[Data_Consolidation|Data consolidation]] physically brings data together from several separate systems, creating a version of the consolidated data in one data store. Often the goal of data consolidation is to reduce the number of data storage locations. Extract, transform, and load (ETL) technology supports data consolidation.
 ETL pulls data from sources, transforms it into an understandable format, and then transfers it to another database or data warehouse. The ETL process cleans, filters, and transforms data, and then applies business rules before data populates the new source.
-*Data Propagation: [[Data_Propagation|Data propagation]] is the use of applications to copy data from one location to another. It is event-driven and can be done synchronously or asynchronously. Most synchronous data propagation supports a two-way data exchange between the source and the target. [[Enterprise_Application_Integration_(EAI)|Enterprise application integration (EAI)]] and [[Enterprise_Data_Replication (EDR)|enterprise data replication (EDR)]] technologies support data propagation. EAI integrates application systems for the exchange of messages and transactions. It is often used for real-time business transaction processing. Integration platform as a service (iPaaS) is a modern approach to EAI integration. EDR typically transfers large amounts of data between databases, instead of applications. base triggers and logs are used to capture and disseminate data changes between the source and remote databases.
+*Data Propagation: [[Data_Propagation|Data propagation]] is the use of applications to copy data from one location to another. It is event-driven and can be done synchronously or asynchronously. Most synchronous data propagation supports a two-way data exchange between the source and the target. [[Enterprise_Application_Integration_(EAI)|Enterprise application integration (EAI)]] and [[Enterprise_Data_Replication (EDR)|enterprise data replication (EDR)]] technologies support data propagation. EAI integrates application systems for the exchange of messages and transactions. It is often used for real-time business transaction processing. [[Integration_Platform_as_a_Service_(iPaaS)|Integration platform as a service (iPaaS)]] is a modern approach to EAI integration. EDR typically transfers large amounts of data between databases, instead of applications. base triggers and logs are used to capture and disseminate data changes between the source and remote databases.
 *Data Virtualization: [[Virtualization|Virtualization]] uses an interface to provide a near real-time, unified view of data from disparate sources with different data models. Data can be viewed in one location, but is not stored in that single location. [[Data Virtualization|Data virtualization]] retrieves and interprets data, but does not require uniform formatting or a single point of access.
 *Data Federation: [[Data Federation|Federation]] is technically a form of data virtualization. It uses a virtual database and creates a common [[Data_Model|data model]] for heterogeneous data from different systems. Data is brought together and viewable from a single point of access. [[Enterprise_Information_Integration_(EII)|Enterprise information integration (EII)]] is a technology that supports data federation. It uses data abstraction to provide a unified view of data from different sources. That data can then be presented or analyzed in new ways through applications. Virtualization and federation are good workarounds for situations where data consolidation is cost prohibitive or would cause too many security and compliance issues.
@@ Line 49: / Line 49: @@
 '''Data Integration Process'''<ref>What are the Essential Steps in the Data Integration Process? [http://athena-solutions.com/essential-steps-in-the-data-integration-process/ Athena]</ref><br />
-People usually oversimplify data integration by assuming it involves only [[Extract,_Transform,_Load_(ETL))|extract, transform and load (ETL)]] tools. Though critical, an ETL tool is just one piece of a complex puzzle. The [[Data_Integration_Framework_(DIF)|data integration framework (DIF)]] encompasses two categories of processes. The first category is the process to determine your data requirements and solution. The second is the process used to physically gather the data from its sources and transform it into information that businesspeople can use to analyze and make decisions.The more attention paid to the processes, the more robust, maintainable and understandable the data is both to IT and businesspeople.
+People usually oversimplify data integration by assuming it involves only [[Extract,_Transform,_Load_(ETL)|extract, transform and load (ETL)]] tools. Though critical, an ETL tool is just one piece of a complex puzzle. The [[Data_Integration_Framework_(DIF)|data integration framework (DIF)]] encompasses two categories of processes. The first category is the process to determine your data requirements and solution. The second is the process used to physically gather the data from its sources and transform it into information that businesspeople can use to analyze and make decisions.The more attention paid to the processes, the more robust, maintainable and understandable the data is both to IT and businesspeople.
 *Determining Data Requirements (At the end you will know what data to source and store)
 **Gathering business requirements