Tips on How to Solve Your Data Challenges

Picture this. You’ve finally secured that all important meeting with the boss to show them this new wham-bang demo software tool that you’re certain is going to solve ALL your companies’ problems!!

While the boss sits patiently waiting for you to get started, you pull up the browser with the software application only to discover that there’s no data, or the data suddenly doesn’t look right. Items are not located where they should be and even worse - the values are wrong. You panic…

Realising that everything has all gone ‘pear shaped’, you calmly make up the excuse that something strange has happened and would they mind if you reschedule.

As you rapidly retreat from the meeting, you’re on the phone to IT who then work furiously to get the data issues resolved. This usually starts by manually updating all the records in the database.

But is manually updating the records going to solve the issue?

Your IT team reassures you that all is not lost and they will resolve the problem with a series of minor (but manual) tweaks and fixes. But the opportunity to showcase your diligent efforts in front of the boss has now passed and who knows when you’ll get another appointment. What a nightmare!

Does this sound familiar?

In my 30-year odd experience as a CIO/CTO, I’ve found that the underlying problem is actually with the data and no amount of manual intervention is going to cure it. Too many times have I seen IT create one more ‘short-cut’ or add one more ‘exception’ and then say everything will be fixed. But everyone knows you can never fix a hole just by digging deeper! This just adds even more complexity to your dodgy datasets.

What was clean data to start with has now turned into dirty data due to the many past integration projects, syncs, “minor fixes” and feed adjustments. The data has lost its value. What a waste…

So, what should you do about it?

I don’t mean to sound all doom and gloom because in reality there are things you can do to keep on top of your data. Here’s a few ideas, but I would welcome more suggestions in the comments section.

1. Allocate a dedicated data analyst or team

Someone whose job is to cleanse, de-duplicate and reconcile your most complex feeds and data sources. Typically the most business critical ones are ERP with the financial data and the CRM with customer data. These are the lifeblood of any organization. Keeping these two systems clean is a seriously big advantage (you’d be surprised at how easy this sounds, yet how uncommon it is).

2. Take an inventory of your IT systems

Without trying to boil the ocean with yet another massive integration project, are there any systems you can easily dispense with? System rationalization is an issue that is particularly pertinent to organizations that have made acquisitions and have accumulated dozens of stand-alone systems. Decreasing the number reduces the complexity of the network and consequently the data feeds.

3. Focus on the core data interfaces

Much like the human body, your IT infrastructure needs to work together to ensure homeostasis is maintained. By focusing on the core interfaces, which is where blockages are likely to happen, you may uncover issues that affect everything else in the organization’s IT ecosystem and that can be easily corrected.

4. Control the input of data (where possible)

Take an audit of some recent data input entries. Every bit of data comes from somewhere and in spite of so many technological advances, in many organizations today there are still a lot of people responsible for data entry. As you’ll hear me say many more times, anytime something is manual, it becomes prone to human error. To avoid these “carbon-based” errors keep an eye on this, and schedule regular training sessions.

Even if you think everything is working ok, unfortunately the reality of IT is that most times it’s not and needs constant monitoring. I strongly encourage any organization to future-proof your IT systems by starting with your data.

After all as Marc Andressen’s quote goes “Software is eating the world”, and there’s only going to be increasing demand to drive the efficiencies which only software can offer. Without clean data, software’s ability to make a tangible impact on your organization will ultimately be nil.

If you're contemplating building your own software, click on the link below to download our guide covering the pros and cons both in-house builds versus buying an outsourced solution.