Last week, a long-time Informatica customer told me about how he is managing change in his data integration environment on a truly massive scale. The story he told reminded me of someone doing a heart transplant on themself while running a marathon. … blindfolded.
- First, he is delivering several innovative new applications. One to improve customer service. Another, proactively monitors operational performance and suggests where the business needs to invest more to improve their capacity and service levels.
- Next, he is rolling out new systems that will collect customer sentiment analysis from social media sources (Big Data) and integrate that with ongoing campaigns and company planning.
- On top of that, he is managing the data integration aspects of a galactic merger between his company and another large company. In this effort, he has to manage application consolidation, application retirement, data migration, and data integration projections between the two companies. The merger has to be accomplished while keeping everything up and running.
- Finally, he also has to deal with a complex and changing regulatory compliance environment that makes demands in terms of security, privacy, and financial reporting and auditing.
It’s a challenge to manage change on that scale, and at the same time, to maintain the data integrity of all of the enterprise software applications . But, most Data Integration managers I know are doing exactly that without the visibility necessary to effectively manage change on that scale. Managers worldwide are struggling to manage change in complex Data Integration environments where a seemingly simple change has huge business risk and can result in material negative impacts. The problem is that multiple systems may have cross-dependencies on the data objects that are being changed. A change to one data object cans cause ripple effects across the system, impacting multiple applications, reports, and dashboards. A loss of data integrity in any of these could lead to bad business decisions costing millions.
Examples of Data Integration Environment Changes
Common examples of changes to the data integration environment that can negatively impact data integrity include:
- Business Efficiency Initiatives – Such as, application virtualization, application consolidation & migration
- New Business Initiatives – Such as, new products, new offers, M&A activity
- New Data Types – Such as, cloud data, third party data services, social media data
The list goes on, but each change has a web of interdependencies that carries a level of risk to the business.
Managing Data Integration Change With Metadata
When Data Integration change requests have the potential to impact the quality of the data underlying key business decisions, it’s critical to have the right tools to manage that change process.
Metadata management tools provide the ability to capture the metadata in your data integration environment and to give you the tools to visualize, manage, and control that change.
The key capabilities of good metadata management tools such as Informatica Metadata Manager include:
- Visibility: A visual map of the data flows within your Data Integration environment.
- Data Lineage: The ability to trace data from source-to-target within the Data Integration environment and they ability to inspect transformations, business rules and reference data.
- Impact Analysis: The ability to see what other data objects upstream and downstream of the data object you are changing would be impacted by that change.
- Metadata Connectors: These automatically collect and present the technical metadata found in data integration tools, applications and repositories, data warehouses, Business Intelligence tools and data modeling tools.
For more information on managing change, see the Informatica Chalk Talk Video: Metadata Management to Reduce Delivery Cycles
One question I hear a lot is: “Is this the right time to make the investment in a technical metadata management project?” Before responding, there are a couple of questions to consider first:
- What is the likelihood of a change to your Data Integration environment causing bad data?
- In your environment, what is the cost of a bad business decision?
- How important is it that you able to “prove” your numbers to auditors?
Increasingly, the data is the business. Your business is only as good (and as competitive) as its ability to manage the integrity of the data it uses or provides. Nobody should perform heart surgery blindfolded, and nobody should make changes to their data integration environment without the tools to give them proper visibility either.
For more information, see the previous metadata blog: “Who Needs Metadata?”
