SAP Datasphere Data Integration - Part 1 - Introduction and Integration using Remote Tables

SAP Datasphere provides a large set of default connections to access data from a wide range of sources, source from SAP as well as non-SAP sources or partner tools, residing in an cloud as well as on-premise environment.

Connections are defined in a separate view which can be found in the main navigation window of the left. Connections are individual objects in SAP Datasphere. They are created and maintained per SAP Datasphere Space which means only members of a specific Space are able to make use of the related connections.

These connections can be then be used within the different tools (like SQL View Builder, Graphical View Builder or Data Flow) for creating your data models.

SAP Datasphere can integrate with both SAP or Non-SAP Source systems and it can be either cloud or on-premise.

We have multiple connection types in SAP Datasphere. Each connection type supports a defined set of features. Depending on the connection type and the connection configuration, a connection can be used for one or more of the following features:
  1. Remote Tables - helps building views. During import, the tables are deployed as remote tables. Depending on the connection type, you can use remote tables to: 
    • directly access data in the source (remote access)-
    • copy the full set of data (snapshot or scheduled replication)
    • copy data changes in real-time (real-time replication)
  2. Data Flows - After you have created a connection, in the data flow editor of the Data Builder, a modeler can add a source object from the connection to a data flow to integrate and transform your data
  3. External Tools - SAP Datasphere is open to SAP and non-SAP tools to integrate data to SAP Datasphere
  4. Model Import - The model import is a special feature for SAP BW/4HANA or SAP S/4HANA Cloud as a source. It supports importing meta data having to rebuild them manually. After you have created a connection, from the entry page of the Business Builder, a modeler can import source meta data from the connection
Integration Architecture:
SAP Datasphere leverages different technologies to setup connection to sources. Broadly there are three type of Integration possible:
  1. Remote Tables
  2. Data flows
  3. External Sources

Overview of Data Integration Approach:

Integration using Remote Tables:

Most of the SAP sources are able to integrate as Remote tables in Datasphere as sources. In for this SAP Datasphere data integration approach, the SAP HANA Federation Framework is currently mainly based on SAP HANA Smart Data Integration (SDI) and its data provisioning framework (dpServer + dpAgent). There are some sources which are able to setup a direct connection (e.g. SAP SuccessFactors, SAP HANA Cloud) or SAP HANA Smart Data Access (SDA) with the Cloud Connector (e.g. SAP HANA on premise). But in fact, most SAP Datasphere connection types that support creating views and accessing or replicating data via Remote Tables for this purpose leverage this so-called SDI Data Provisioning Agent (DP Agent). Appropriate Setups are needed before creating any connections.

What are DP Agents?

DP Agents is a light weight component running outside the SAP Datasphere environment. It hosts data provisioning adapters for connectivity to remote sources thus enabling data federation and replication. This basically acts as a middleman (gateway) to SAP Datasphere providing secure connectivity between the database of your SAP Datasphere tenant and the adapter-based remote sources. These are managed by DP Server. Typically this is needed by all the SDI connections.

Through the Data Provisioning Agent, the pre-installed data provisioning adapters communicate with the Data Provisioning Server for connectivity, metadata browsing, and data access. The Data Provisioning Agent connects to SAP Datasphere using JDBC. It needs to be installed on a local host in your network and needs to be configured for use with SAP Datasphere.

Now that we understood what are dpAgents and how it helps in SDI connection to SAP Datasphere for Remote tables, we must understand how SDA Connections are done in SAP Datasphere for Remote tables.

Only for SAP HANA on-premise based Remote tables, SDA connections are used. These are done using a Cloud Connector as a link between SAP Datasphere and SAP HANA on-premise.

What are Cloud Connectors?

The Cloud Connector serves as a link between SAP Datasphere and your on-premise sources and is required for connections that you want to use for following use cases:

  1. Data flows
  2. Model import from SAP BW/4HANA Model Transfer connections (Cloud Connector is required for the live data connection of type tunnel that you need to create the model import connection)
  3. In rare cases also for Remote tables: Only for SAP HANA on-premise via SDA

The above picture would provide an idea of SAP Datasphere Connection types that support Remote table Federation.

Data Access (Remote/Replicated):

By default, when you import a remote table, its data is not replicated and must be accessed using federation each time from the remote system. 

You can improve performance by replicating the data to SAP Datasphere and you can schedule regular updates (or, for many connection types, enable real-time replication) to keep the data fresh and up-to-date. 

During Replication, data is replicated physically from the source into SAP Datasphere. 

  • For snapshots, replicated means that data is read from the replica table which is not updated in real-time. 
  • For real-time replication, replicated means that data is read from the replica table and expected to be updated in real-time if changes occur in the original system.

Virtual access is also possible using remote tables, that points to a table in an external system without copying the data. So, the data is transferred via network each time a query is executed through which the data is accessed directly from the source and read from the virtual table.

Data Integration from SAP BW Bridge as Remote table:

A dedicated connection called BWBRIDGE of type SAP Datasphere, SAP BW Bridge is available to import SAP BW bridge objects into SAP Datasphere as Remote Tables. This SAP BW bridge connection differs from other connection types, as it can't be created or modified. It is being generated by default by SAP when the SAP BW bridge component is provisioned.



Comments

Post a Comment

Popular posts from this blog

SAP Datasphere Architecture

Image
  SAP has retained all of the functionality of SAP Data Warehouse Cloud (SAP DWC), while adding new functionality that improves the discovery, modelling and distribution of enterprise data. This evolution has had no impact on the workload of SAP DWC customers. The update is performed automatically, so no migration is required. This will be the case for all new functionalities Accessing and using diverse data from different information systems and locations such as cloud providers, on-premise systems and different data sources has always been a complex challenge for organisations. The processing that this data undergoes in order to centralise it and return it afterwards (extraction, transformation, export, etc.) leads to the loss of its business context. SAP Datatsphere gives its customers the benefit of an enterprise data fabric architecture that harmonises critical data (SAP and non-SAP) in a single way and rapidly delivers meaningful data with business context and logic. SAP Datasphe
Read more

SAP Datasphere Spaces

Image
  A space is a secure area created by an Administrator, in which members can acquire, prepare, and model data. The Administrator allocates disc and in-memory storage to the space, set its priority, and can limit how much memory and how many threads its statements can consume. If the administrator assigns one or more Space Administrators as members of the space, they can then assign other members, create connections to source systems, secure data with data access controls, and manage other aspects of the space. Spaces are virtual work environments with their own data bases. Spaces are decoupled, but are open for flexible access, thus allowing your users to collaborate without having to worry about sharing their data. To model your data and create stories, you need to start off with a space. You can decide how much and what kind of storage you need, as well as how important your space is compared to other spaces. You also add space members and set up connections here. Each spaces can be
Read more