Cloud and On-premises Integration Tips

It’s no secret that the adoption of cloud services is rapidly increasing and that most enterprises have already deployed line of business applications in the cloud, especially for CRM solutions such as Dynamics 365, Salesforce, Sage CRM and Infor CRM.

Software as a Service (SaaS) and Infrastructure as a Service (IaaS) are now widely accepted as important additions to the enterprise infrastructure environment and survey results (as in the Gartner graph below) show that this trend is expected to continue.

Despite this shift, most enterprises still have substantial local data repositories and applications for a variety of reasons, including:

• many legacy applications are not good candidates for moving to the cloud
• databases may be embedded into enterprise work flows in ways that make moving a difficult and time consuming affair
• data may be too sensitive, or there may be legal restrictions on deployment.

It is too early to say definitively what the balance of cloud versus non-cloud will be long term. The general expectation is for on-premises solutions to all but phase out, however there is also the view that certain applications will remain on-premises for reasons of speed, efficiency or security and that cloud adoption will plateau. Either way, hybrid (public, private and on-premises) environments are likely to be in place for many more years as the table below indicates.

Source: Worldwide Cloud IT Infrastructure Market Forecast, Q4 2016

Although a hybrid setup is common it creates difficulties when data is moved or merged between the different environments, difficulties so challenging that integration with existing architecture is currently seen as the biggest barrier for cloud adoption (KPMG, Journey to the Cloud, 2017). A large part of this is due to performance. This is because a cloud system, by definition, is not on a company’s internal network but on servers accessed via the internet, typically via a web service API of some kind or a browser for user interaction. As web service API calls are orders of magnitude slower than interacting directly with a database, response times can be dire.

Overcoming these performance challenges is something my company, InaPlex, frequently manages when integrating on-premises systems with cloud services such as Salesforce or Dynamics 365. Our integration solution, Inaport, has a range of techniques and functionality that can optimize performance between on-premises and cloud, including specialized matching, local caching, and local cross-reference tables. These areas are described in more detail below.

Standard matching techniques speed integration tasks

Inaport has a range of matching techniques, but the easiest to use and in many ways the most powerful, is Standard Matching.

Standard Matching allows you to build a match on any field or combination of fields on the target system, and to use expressions on the field values. For example:

 normcomp(#name) & #address1_postalcode 

will normalize the company name and append the postal code.

When the job starts, Inaport will build a local in-memory index on the target entity, and then use that local index for all matching. The advantage of this approach is there is only a single web service call to the target system instead of a web API call for each source record. This can cut the number of calls by thousands, with corresponding improvements in speed.

Local Caching

More complex integrations will frequently need to query the target system based on data in the import. For example, there may be a requirement to look up product, currency, or some custom entity related to the incoming data to supplement, or make decisions.

If you have to query the target for each incoming record, performance will suffer significantly. Instead, Inaport provides the option to create local in-memory caches of required data from the target. You can create as many of these caches as required, and then query the cache rather than the target.

As with Standard Matching, the advantage is there is a single call to the target system, and you can pull only the data required. In addition, you can decide at run time whether you need the cache. If the incoming data does not require it, you can elect to not query at all.

Local Cross Reference Tables speed mapping between on-premises and cloud.

A different technique for local caching is to build and update cross reference tables on the local system during an import. A cross reference table typically holds primary key or matching data from both the source and the target system.

The advantage of this approach is that when querying the source data you can join with the cross reference table to get the primary key of the target system. Inaport then supports using that primary key as the match criteria – with zero cost for matching.

Inaport supports building the cross reference table in two ways:

1. after an entity is created/updated in the target, Inaport provides the primary key of that entity or any parent entity
2. that data can then be used in post-operation expression to update the local table, either in the source database or any other database.

Summary

While the move to cloud-based solutions will continue apace, most enterprises will retain at least some on-premises systems for the foreseeable future, and these will continue to require integration with the cloud. Managing the hybrid infrastructures will remain challenging, especially in terms of performance, however there are techniques and tools which can provide a dramatic improvement in speed and efficiency, as this post has described.

At the end of the day, the priority is to ensure your company gets the most out of its digital information, wherever it resides. This is an area in which InaPlex has considerable expertise, and we are happy to help out with any advice, instruction or assistance you may need.

InaPlex Inc. specializes in systems, solutions and services for CRM integration. Request a free consultation or webinar at Contact InaPlex.