Data Integration: Cloud and On-Premises Solutions

In a recent G2 Crowd survey (May 2016, N = 347), 62.2 percent of the respondents reported that data integration across multiple platforms was having an impact / high impact on their current business strategy. That answer was a tie for second in the overall survey for impact on strategy, with only improving customer experience scoring higher. As information becomes more of the driving force behind business strategies and business decisions, getting accurate and complete data to the right person at the right time and in the right context is a critical business imperative. You could even make a good argument that many of the other important business issues are in some ways linked to the lack of integrated data. The number one answer on the survey, improving customer experience (72.2 percent) has several ties to lack of integration, including the difficulty of simply providing a complete view of the customer, and having the ability to consistently interact with them across all of the potential interaction points.

Not that this is a new topic though, it’s an issue that businesses have dealt with for decades. Today though, with the rapid increase in the amount of data, the increasing amount and types of technology used by businesses and the complexity that is introduced by silo’ed systems, multiple deployment models, diverse architectures and even the proliferation of data locations and endpoints that need to consume the data, the problem seems bigger and more messy at the very time solving it would have the most significant impact for many businesses. The causes of the huge amount of data fragmentation in most businesses are diverse but include:

  • Proliferation of line of business driven point solutions, which create data silos: This is caused by several factors, line of business executives are empowered to solve their technology problems and sometimes go around IT, cloud applications can often be deployed with little technical support, and there are many more cloud based point solutions available tied to very specific modern business problems.
  • Hybrid systems: Many businesses are in the long process of shifting from on premises systems to cloud based systems and are currently operating a hybrid model with some mix of public and private cloud and on premises applications. Economics dictate that systems be used until there is a compelling business case for the replacement, and often that replacement is done in a more granular approach of replacing modules instead of complete systems. Modernizing by module leads to hybrid deployments since most of the most modern software products are only available as cloud applications.
  • Data velocity and size: I could quote study after study on the predictions of data growth over time, but a simple Google search will easily answer that question. I’ll just add here that the creation of data has, and continues to accelerate exponentially, which then obviously drives this massive increase in data over time.
  • Integration is still expensive and complex: Maybe I should say it as “integration still isn’t cheap”, but either way, integration requires investment in time, tools and people. The other end of that though, is the complexity issue. I suppose integration almost always has an element of complexity but in a hybrid environment with diverse systems and deployment models, even with SOA, it’s not easy.
  • Organizational silos feed into data silos: I feel like this topic is as old as business but it is very relevant. One of the biggest hurdles to integration is the organizational silos that drive systems silos and of course, system silos = data silos. This is made somewhat worse by the first point on proliferation of point solutions.
  • M&A activity often adds to system and data complexity: This is also pretty obvious, the more new or unintegrated systems that are introduced the more the data gets fragmented and stranded in all sorts of places. I remember a discussion with a CIO of a manufacturing company, who was asking for advice on app modernization. Her biggest challenge was all of the systems that had been added to the app portfolio through multiple acquisitions over the past few years. When I asked her how many ERP systems she was trying to consolidate and modernize she surprised me and the whole room with a whopping 36 different ERP systems…and nearly half of them were different major releases of the same system. None of them were integrated and varied between monthly or quarterly functions.
  • Structured and unstructured data and the massive growth in sources of data: Diversity of data and diversity in sources of data continue to increase. The internet as an enabler ensures that more data and data sources will continue to be defined and with the addition of Internet of Things (IoT) sensors adding new data sources rapidly, the rate of acceleration continues to increase.

That’s some of the causes to the problem, but how do you get past them and approach integration. Here are a few things to consider:

  • Fix your organizational issues / silos and you will at least be able to build an integration plan that has some potential for success and is much more likely to include all the integration needs.
  • Consolidate systems and data sources as much as possible. The consolidation efforts can (and often is) be a part of a broader application modernization program or can be it’s own project (and in some companies it’s a massive project).
  • Refresh the IT line of business partnership to get solution purchases tied to an overall plan, strategy and platform. I’m not suggesting to slow down the efforts of the business, on the contrary I’m suggesting that IT be an integral part of that agile approach to problem solving.
  • Have an integration plan and use it. (I realize that sounds so simplified to border on stupid, but you’d be surprised, or maybe not, how many businesses I talk to that don’t have any integration plan at all)
  • Focus on customer and employee data first. To refer back to the survey, the number one business issue that was impacting strategy was customer experience and the number four (close behind integration and getting better leads, both at 62.2 percent) was finding and retaining the best employees (61.9 percent).
  • Find the “right” integration tools for the job at hand. That’s where G2 Crowd can help, by providing a view into the actual users of the tools. Here are a few of the top rated integration tools (note: based on G2 Crowd score):

Top 5 Cloud Integration Solutions:

  1. Zapier (score of 70)
  2. Jitterbit (score of 64)
  3. Microsoft Azure BizTalk Services (score of 57)
  4. Dell Boomi (score of 52)
  5. Scribe Online (score of 51)

Top 5 On Premises Solutions: 

  1. Microsoft SQL Server Integration Services (score of 92)
  2. SoftwareAG webMethods (score of 62)
  3. Informatica PowerCenter (score of 54)
  4. Mulesoft ESB (score of 48)
  5. Scribe Insight (score of 46)