I like to compare the inception of a regulatory project with the fuelling of an aeroplane. You know where your destination is and you can work out how much fuel you need to get there, but would you ever think of fuelling the plane with the minimum amount required to get you to your destination? Imagine if you flew into difficulties, had a stronger head wind than predicted or couldn’t land at the planned airport. You would always want to have enough fuel to cover all eventualities, as the last thing you’d want to happen is to fall short (literally). However, as with aspirational projects, you don’t want to carry far too much fuel as this would cause it’s own problems. So you need to find the right balance of risk and reward.

The same goes for projects; if you decide that your destination is the closest possible place that would make you compliant and fall short then you are potentially in a heap of trouble. Likewise, should the regulator change the goalposts and increase the requirements for compliance, you could find yourself having a major re-planning exercise mid-project. So how do you find the ‘happy place’ between aspiration and falling short?

I think focusing only on the minimum is a very pessimistic, short-sighted and unambitious approach for an organisation to take. It’s fine to consider it a stop-over on your journey but please don’t make it your final destination and be sure to have enough fuel (be that budget, resource or strategic vision) to get you where you want to be. Nicola Askham recently wrote about this situation in her paper for IRM. In it she recommends taking a phased approach where the first phase should deal with achieving the foundation of compliance and subsequent phases used to build on this foundation. So again to reinforce the point, the minimum may well be part of your journey where you can stop and refuel, but it should not be your final destination. Breaking the journey into smaller steps allows you approach it in a much more manageable way and gives you natural points to reflect on where you have come from and where you are heading next. That way if you do find yourself having to replan your route, it is much more easy to do so.

From my point of view, 2011 was a pretty good year all things considered. There were plenty of ups and downs, both from the personal and professional aspects of my life but I think, in retrospect, the downs were outnumbered by the ups.

I spent the entirety of 2011 working with a single organisation on their Solvency II programme and although challenging, this gave the year a degree of stability and consistency. In the latter part of the year I began to focus on improving my professional network and as a result I have met (albeit in most cases, virtually) some fantastic people and found the DQ community to be extremely talented, friendly and welcoming. I published my blog and stepped up my use of Twitter and I now feel that I am starting to make some progress with regards to marketing my professional services.

So what do I want to focus on in 2012?

• Learning - I already use every opportunity to expand my knowledge of all things DQ as well as improve my knowledge of related business and technical subjects, so 2012 will be no different in this respect. However this year, I may also be tempted to consider a formal accreditation to help improve my credibility to clients and help bolster my own self-belief. The wealth of information and knowledge sharing by the online communities these days is truly priceless and it means you always have access to high quality  learning materials.
• Experiencing - I stated in my DataQualityPro interview last year that I was keen to increase my experience in other industry sectors and this is something I’d certainly consider this year. Although I have been able to move from banking in my previous role to insurance with my current role, I’d be interested to experience life outside of the financial sector.
• Networking - I plan to put effort into expanding my network this year and will try to participate and contribute to the online community more. The rise in social media has been a massive advantage for me and I think anyone not involved in building an online presence (or at least benefiting from the vast amount of online resource) is really missing out.
• Building - I want to spend some time working to improve my own ‘toolkit’. By ‘toolkit’ I mean my own collection of processes and checklists for things like client engagement, data profiling, stakeholder management etc. Also, all the good reference materials like generic DQ frameworks, Governance models and various templates for standard documents that I can dip into and will allow me to hit the ground running with a new client.
• Living - Very important to me is to ensure that I am always in a position to maintain a healthy work/life balance.

Here’s to a great quality 2012!

It’s not that it’s a difficult question to answer (although attributing costs and impacts can be challenging) it’s just that I can’t help but find this question at times a lazy and ignorant response to the raising of a data related issue. It often belittles a problem and inadvertently endorses a culture that is largely indifferent to data.

Maybe it again comes down to the fact that data is not tangible and that senior management, as a general collective, are not culturally or practically used to thinking about data related matters. (Perhaps this will change in time, as there is a gradual generational shift)

What reaction would follow news related to a tangible (and more easily valued) asset? Say there was a fire in a property; I doubt that the immediate response would be “so what?” It would more likely be a machine-gun set of questions: Was anyone hurt? How bad was it? Is it still operational? What’s the impact? What’s the cost to repair? Does our insurance cover it? How did it start? How do we stop it happening again? Have we checked all our other properties?

All valid questions and all equally valid in relation to a data related issue, so why the difference in reaction?

It could of course be down to the way the issue is presented. The challenge is in the common default fascination for managers to have endless metrics and evidence without the understanding of what it is telling them. As I stated in my previous post, there is a natural tendency to take data for granted and by focusing on the minutiae of metrics people can get lost and confused and consequently miss the point; cue the “so what?”

You need to force people to step back for a minute and think about things more conceptually; step back to a level where you are sure they understand the impact and then you can bring it slowly forward again in order to properly quantify and analyse the issue. As with the fire there will be warning signs: the smell of burning; or the sight of smoke long before you feel the heat. Of course it would be easier if management could also understand these signs, but they often don’t and therefore the controls and alarms you put in place need to clearly convey the danger in a way that is understandable to all.

We need to be able to turn the “so what?” into the “tell me more”.

Any mature organisation that uses data will know that, data can simultaneously cost you money and save (or make) you money through-out it’s lifecycle. The enduring challenge is ensuring that you can overcome the associated costs and tip the balance in favour of the saving/making money side. Danette McGilvary states, in her book, that all stages of the data lifecycle have associated costs and by continuing the illustrative analogy of balancing a set of scales we can begin to show what factors in this lifecycle can affect the equilibrium.

On one side of the scales you have the costs associated with things like data storage, software/hardware costs, maintenance overheads and the cost of resources required for data capture. In order to counter-balance these costs we need to ensure that the other side of the scales hold enough weight and we can establish ways to actually reduce weight on the costs side.

As part of your balancing act you need to ensure that your data assets are appropriately managed throughout the data lifecycle. This will involve finding ways to reduce storage (think MDM and SCV), reducing the number of occasions that the obtaining of data is duplicated across the organisation and most importantly use and reuse the data more. A key tactic will be to unearth new stakeholders and new uses for data assets.

Conceptually it is easy to see how you can balance cost and savings but actually working out what the values on both sides are can be incredibly difficult. Tangible costs and savings are easy to quantify however it’s the unseen costs and savings that can often tip the scales one way or the other. For example, until now I haven’t even mentioned the quality of your data; we’ve assumed that the data is of sufficient quality to help tip the scale but what if you have quality problems?

As you’d expect poor data will not help you save money; it will in fact tip the scales the other way. In the same way that a defective component will impact the quality of the end product in a manufacturing process, poor data will impact outputs and cost you additional funds to correct. The more you reuse poor quality data, the more it costs you in the longer term. All those ‘shadow’ processes that have secretly crept into business areas in order to fix or compensate for bad data cost you and ultimately impact the actual value of your data assets. Your data has a potential value that can only be realised when the organisation is using and managing it in an optimal way.

This post obviously tries to very briefly cover a very small part of a very large subject that would easily fill a book (and indeed has filled many), but I thought it would be useful to start the topic and maybe revisit it with later posts. I’ll leave you with some basic concepts to think about:

• Good quality data saves or makes you money every time it is used (assuming appropriate use).
• Bad quality data costs you money every time it is used.
• Each stage in the data lifecycle costs money. Try to optimise and reduce duplication in each stage.
• Promote and increase the use and reuse of the same data across the organisation.

As always, thoughts and comments encouraged.

I’m a big fan of ‘not reinventing the wheel’, when there is no real need to, and always look for methods and tools that have been successfully utilised by others in the past. Having previously read Arkady Maydanchik‘s book Data Quality Assessment, I was keen to use the same rule categories that he proposed. For those that haven’t read Arkady’s book I’ll note the categories and sub-categories that he suggests but for the more detailed explanations I’d recommend buying Arkady’s book (I’m not on commission, I promise) or checking out his articles at DataQualityPro.com and IAIDQ.

Arkady suggests that DQ rules should fall into one of the five broad categories outlined below:

1. Attribute Domain Constraints
2. Relational Integrity Rules
3. Rules for Historical Data
4. Rules for State-Dependent Objects
5. General Attribute Dependency Rules

Within each of these categories are more specific sub-categories that allow more precise categorisation of rules (there are also sub-sub-categories, but I’ll leave them out here):

1. Attribute Domain Constraints
• Optionality Constraints
• Format Constraints
• Valid Value Constraints
• Precision Constraints
2. Relational Integrity Rules
• Identity Rules
• Reference Rules
• Cardinal Rules
• Inheritance Rules
3. Rules for Historical Data
• Timeline Constraints
• Timeline Patterns
• Value Patterns
• Rules for Event Histories
4. Rules for State-Dependent Objects
• Domain Constraints
• Transition Constraints
• Timeline Constraints
• Advanced State Dependant Rules
5. General Attribute Dependency Rules
• Attribute Redundancy
• Attribute Dependency
• Partial Dependency
• Attribute Correlation

What you should find is that any DQ rule you can come up with, fits nicely into one of the above categories. This therefore allows you to group rules together in a more meaningful way and, post data assessment, helps you formulate the grouping of erroneous records for reporting and remediation.

Cataloguing the rules in this way also ensures that you (and your colleagues) are working within a structured and consistent framework that you can use as a reference when defining DQ rules in the first place. It is important to use it as a kind of aide-mémoire to ensure that you are considering all categories for each data element.

Lastly, I would also recommend having your own specific examples for each category of rule. I have found that this becomes an invaluable training/education/communication resource to have at your disposal.

Happy cataloguing!

For those of you who haven’t seen an episode of ‘House’ I’ll provide a generic plot breakdown:

• Someone is brought to hospital with an unknown or rare ailment.
• Dr Gregory House is brought in because no one knows what to do; his skilled (and long-suffering) diagnostics team detail the symptoms to House and offer a diagnosis (usually lupus).
• House is rude to everyone and makes a flippant diagnosis of his own.
• Everyone disagrees with House but they start treatment anyway.
• The treatment doesn’t work and the patient gets sicker. House gets angry and his diagnostics team suggest a number of other diagnoses.
• House is rude to more people and ignores all other diagnoses.
• In the middle of insulting someone House has a moment of clarity/inspiration and comes up with the correct diagnosis.
• Treatment is given just in time and patient recovers.
• House is rude some more. End credits.

Ok, so where am I going with this? Well I’m certainly not proposing we all adopt House’s method of ‘working’ but I do like the concept of the dedicated Diagnostics Team. This team is highly skilled, highly intelligent and can be called into action whenever the need arises.

Would it be beneficial for an organisation to support a similar type of team as part of a data quality programme? A small team of highly skilled individuals to act as DQ subject matter experts and act as a shared resource across the organisation. Certainly I can see obvious benefits to a relatively immature (in DQ terms) organisation having a team of subject matter experts ‘on call’ to assist with data related projects. A small core team is cheaper to run and will have more immediate impact than trying to spread skilled resource across many business areas. It gives the organisation something to build on and allows them to focus effort where and when it is required.

Of course there are pitfalls and occasions where this just can’t work practically but surely it is still something worth considering? I just recommend that you employ someone slightly more approachable/manageable/polite than Dr. Gregory House.

• Identify risks and associated impacts. Executives live in fear of risks that they haven’t considered and mitigated. Point out the data related risks and explain how to handle them. Be sure to have facts and real examples to back up your points. Use previous data related disasters to your advantage.
• Sell the benefits. Senior management are the key to the success of any initiative and will always want to know where the benefits to the organisation are. Make sure that benefits will out-weigh any associated costs. Try to focus on points related to the reduction of risks and opportunities for revenue increase. Remember that better data = better, more informed decision-making. Use regulatory imperatives and known best practices as leveraging tools when required.
• Don’t treat DQ initiatives as technology initiatives. Data is a corporate asset and needs to be considered as such. Whilst IT should always be involved, it is vital that initiatives are business led.
• Make more data available. The more data available to people and the more people using the data; the more opportunity to uncover problems and increase the demand for better quality data.
• Improve communication and ensure transparency. Have a consistent message and make sure it’s the correct message. Make sure the message is written in business speak. Give people a forum to talk about data and allow them to open up about problems. Ensure there is a non-aggressive and no blame environment even if that means anonymity. Don’t miss any opportunity to spread the word and make findings widely available if appropriate.
• Top-down and bottom-up. Whilst it is important to engage with senior colleagues, it is equally important to identify those people at the coal face. The people who know the data; who use the data; who create the data. Identify and formally recognise existing good practices. Begin to educate and train people at all levels. Promote collaboration to help amass support and create a ‘bigger voice’ to help with senior management engagement.
• Start small but think big. It will always be easier to get smaller initiatives kicked off but ensure they are consistent with the ‘bigger picture’. Make it clear that data quality is not a one-off exercise; it requires a culture change. Organisations need to increase focus on using data and not just producing data.

Obviously there are many, many, more points to make but hopefully I’ve covered a lot of the main ones. If you’ve got 5 minutes I’d recommend heading over to the DataQualityPro.com forum to have a read of the latest replies or indeed feel free to leave a comment here.

It’s a fantastic opportunity for me to reach some new people and hopefully it’s an interesting read too!

I know there is currently a bit of a drive to improve community activity on the site and I hope that more people choose to contribute because the site is a really great resource. So if you aren’t a member over there already please go and sign up and contribute to the many forum discussions.

I was intrigued to hear that a couple of weeks ago, Google finally killed off their ‘Image Labeler’ game/service/tool thingy. I remember back in 2006, when it first arrived as a ‘beta’, thinking it was another piece of genius from those clever chappies at the Googleplex. Managing to subtly get their own customers to improve the quality of their image catalogue’s metadata….for free….by thinly disguising it as a game….Genius!

For those of you who haven’t seen or heard of Google Image Labeler (<– American spelling; but that’s a post for another day) the premise is as follows.

• You are paired up randomly to play against another person online.
• Over the period of the game (usually 2 minutes) you and your playing partner are shown the same set of images. You have to type as many words (labels/tags) as possible to describe the images.
• When one of your words matches your playing partner’s words you both earn points and you move onto the next image.
• More points are awarded for more specifc descriptions. For example the word ‘tree’ will have a low score whereas ‘oak tree’ will score higher. Also, words that have already been matched by other players in other games are generally not accepted. Both these factors ensure that players learn that they need to be more specific with their descriptions (and therefore provide higher quality data).

The premise is so simple; I remember thinking that it must be possible for other organisations to employ a similar technique to enrich their data. Surely an online retailer, for example, could call on their customers to help them catalogue their inventory in a similar way; perhaps offering discount products as a sweetner? Other organisations could reward people for spotting data issues with similar incentives. Obviously there are limitations with regard to the type of data you could open up to the masses. For example sensitive or private data would be seriously off-limits. But hey, this is enriched data for free…right?

What could possible go wrong with such a simple but genius process? Google had their end-users enriching data that would improve their service; had thrown in a bit of fun; and kept people interested by implementing some online rankings recognising the most dedicated players. Oh, wait a second……..what?……..Online rankings? Ah well, that’ll be the problem then.

As anyone with experience of online gaming in any guise will testify; whenever there is a chance people can get their names up in lights, there will be people who resort to dark art of cheating. Sure enough, soon after the new ‘game’ launched, the cat and mouse contest also commenced. Google found that they had to keep implementing controls to stem the flow of poor quality data entering through the very channel they designed to improve it in the first place. Cheaters came up with newer more ingenious techniques for getting good scores by doing none of the good work Google were expecting. (If you are interested in some of these ‘cheats’ then you can read about some of them on the link above or just search on Google !)

Google built this tool based on their expectations related to the natural competitive instinct of their users. Ultimately what happened was that Google actually massively underestimated the competitive desire (or the inherent infantilism) in some people. It’s not abundantly clear whether Google actually benefited in the long term from ‘Image Labeler’ but I’m guessing they did for a while given they let it run for 5 years.

So I guess a question still remains; can you successfully use customer executed data quality initiatives and properly control them? Or will you always have to keep your eyes open for those that will try to beat the system?

I’m assuming you know the events that took place so I won’t go into them in detail again apart from highlighting the heart-wrenching, life or death decision, that Joe’s climbing partner Simon made to cut the rope holding the pair together. Doing so meant Simon could try to get help but also meant that Joe was left to fend for himself. Now the comparison I’m about to make is in no way meant to trivialise such an event, rather I just want to take forward the message that sometimes you need to make difficult decisions in order to move yourself forward and that may well mean leaving something behind.

In the corporate sense, there are a number of internal factors that can slow or hold an organisation back and having worked in the finance industry for over 7 years I can say that issues associated with legacy systems are probably the most painful if not the most common. By legacy, I’m talking about systems developed back as early as the 60′s or 70′s and to put that in perspective, that means running billion pound business using technology that is older than me (yes I am that youthful!). From a DQ point of view these systems can also be a nightmare; from getting data out in a suitable format for analysis to trying to unpick the overloaded and repurposed columns. Even finding someone who knows the system well enough can prove challenging. “Oh yeah you want Dave; he wrote the code for that. Shame he left 15 years ago”. System quirks are usually stuff of legend and are nostalgically excused or laughed about; a bit like that old friend with the weird social ticks that no longer gets invited to parties.

Perhaps more importantly, what I’ve also found is that this dependency on legacy systems makes these organisations very inflexible to even the smallest change. For example it’s like trying to turn a battleship when product development want to launch a new product or someone senior asks a question about the portfolio. God forbid a regulator starts to ask some questions!

So how are these systems still running? Good question. I mean, who these days has a mobile or personal computer that is more than a few years old? I’d like to specifically ask that question to the managers of IT departments; would they be happy running their own affairs on 40-year-old tech? Now granted, the effort to ‘cut loose’ a legacy system is daunting and this in turn put the decision maker in a bit of a difficult position. But what baffles me is the fact that with every passing day these systems become more out-dated and the task of replacing and migrating becomes harder and more expensive. What happens instead is that these out-dated systems are patched and new GUIs are built and eventually most people don’t even realise the old system is still breathing. In real world terms I might as well stick my Granny in a tracksuit and register her for London 2012; it doesn’t mean she’ll run any faster though does it?

So why is it so hard to ‘cut loose’? To me it seems like the more calculated decision to make, but then I’m not currently in a position where I’m holding that metaphoric knife and rope. Maybe it’s because I’m part of the Apple generation where the mantra of smaller upgrades more often becomes the norm.

I’d love to hear your thoughts on this.