Using Data Integration Testing for Reconciling Production Data Assets

In my last post, we started to discuss the need for fundamental processes and tools for institutionalizing data testing. While the software development practice has embraced testing as a critical gating factor for the release of newly developed capabilities, this testing often centers on functionality, sometimes to the exclusion of a broad-based survey of the underlying data asset to ensure that values did not (or would not) incorrectly change as a result.

In fact, the need for testing existing production data assets goes beyond the scope of newly developed software. Modifications are constantly applied within an organization – acquired applications are upgraded, internal operating environments are enhanced and updated, additional functionality is turned on and deployed, hardware systems are swapped out and in, and internal processes may change. Yet there are limitations in effectively verifying that interoperable components that create, touch, or modify data are not impacted. The challenge of maintaining consistency across the application infrastructure can be daunting, let alone assuring consistency in the information results. Read more

Data Governance and Quality: Data Reuse vs. Data Repurposing

February 22, 2012 by · 3 Comments
Filed under: Business Impacts, Data Governance, Data Quality 

I have been assembling a slide deck for an upcoming TDWI web seminar on Strategic Planning and the World of Big Data, and I am finding that I might sometimes use two different terms (“data reuse” and “data repurposing,” in case you ignored the tootle of this post) interchangeably when in fact those two words could have slightly different meanings or intents. So should I be cavalier and use them as synonyms?

When I thought about it, I did see some clarity in differentiating the definitions:

  • “data reuse” means taking a data asset and using more than once for the same purpose.
  • “data repurposing” means taking a data asset previously used for one (or more) specific purpose(s) and using that data set four a completely different purpose. Read more

Prohibition, Mandating “Ethical” Compliance, and Data Quality

February 20, 2012 by · Leave a Comment
Filed under: Data Quality 

I just finished reading a very interesting book on the evolution of Prohibition in the US in the mid-late 1800s and early 1900s. The book, “Last Call” by Daniel Okrent, followed the temperance movement that started with a bunch of men pledging to stop drinking through its alignment with the women’s suffrage movement, to the passage of the Prohibition amendment, followed by its eventual repeal. One revelation to me was that , according to the author, the political processes that enabled the passage of prohibition essentially created the modern methods of political lobbying, the ability of minority parties to significantly sway majority rule, and (when push comes to shove) that when you mandate behavioral changes, you probably should have four things in mind:

  1. Your value proposition must be appealing enough to convince those you are trying to regulate that it is in their best interests to comply;
  2. You should ensure that you have adequate resources for inspection, monitoring, and enforcement;
  3. You don’t allow so many loopholes that enable the ad hoc creation of classes or parties who can blatantly evade compliance; and
  4. You don’t reward illicit behavior. Read more

TDWI Solution Summit: Master Data, Quality, and Governance

December 15, 2011 by · Leave a Comment
Filed under: Data Governance, Data Profiling, Data Quality, Master Data 

I am honored to be one of the co-moderators of the upcoming TDWI Solution Summit on Master Data, Quality, and Governance, to be held March 4-6 in Savannah, GA. I attended one of these events in the past, met a lot of people currently engaged in launching an MDM project, but this year I have helped to find some cases studies that focus on master data as a result of good data quality and governance techniques, not the driver. Please go to the web site and check it out, then apply to be a delegate! Hope to see you there!

Unique Identification, Health Care, Protection of Personal Health Information, Upcoming Webinar

I have a friend in the neighborhood, and coincidentally he shares a name with another person who also lives in our neighborhood. As a joke, while I refer to my friend by his name (say for argument’s sake it is “Arnie Hollingsworth”), I refer to the other guy as “The Other Arnie Hollingsworth.”

Read more

Next Page »