SAP Datasphere - Data Integration - Part 2 - Data Integration based on Data Flows and External Sources

Data Integration based on Data Flows:

A Data Flow is used as per SAP Definition “to move and transform data in an intuitive graphical interface”. The Data Flow is created within the part of the Data Builder in SAP Datasphere, and it currently supports the following operators to be used within your transformation process: 

  1. Join Operator 
  2. Union Operator 
  3. Projection Operator 
  4. Aggregation Operator 
  5. Script Operator 

As we are coming from a strong Business Warehouse background, we are always comparing things to the past. Which in our case would be a classical SAP BW on HANA or SAP BW/4HANA. We currently see the Data Flow as a substitute for what used to be in a BW system the transformation.  


In the data flows, we will be able to use a series of standard transformations without the need of programming knowledge and in a graphic way, but we also have the possibility of creating transformations based on scripts. In a data flow, we will use views or tables that we may already have in our SAP Datasphere or use the connections to get data from other systems, in that case we should first create all the necessary connections in our space.

Steps to be followed to create a Data Flow:

  1. Create Connection
  2. Build Data Flow
  3. Execute / Schedule Data Flow
  4. Data Modeling / Story Creation
As part of Standard transformations, we would combine data sets with no code operators for Projection, Aggregation, Join, Filter, Union, etc. The data sources in this can be tables, ABAP CDS views, OData, Remote Files (JSON, Excel, CSV, ORC). 
As part of scripting, we can do advanced transformation requirements like extraction of texts using the dataflow. Python 3 language is supported in this case. Insert a script operator to transform incoming data with a Python script and output structured data to the next operator. The script operator receives the data from a previous operator. You can provide transformation logic as a body of the transform function in the script property of the operator. Incoming data is fed into the data parameter of the transform function and the result from this function is returned to the output.

Data Integration based on External Sources:


The SAP Datasphere Data builder provides CSV file by default. Once the file is loaded, we can apply various data wrangling and transformation rules such as Concatenate, Split, Extract, Replace, Change or Filter. Beauty of this is a table is automatically created with the derived columns based on the file structure thus allowing consumption of the uploaded data as tables in SAP Datasphere.

Comments

Post a Comment

Popular posts from this blog

SAP Datasphere Architecture

Image
  SAP has retained all of the functionality of SAP Data Warehouse Cloud (SAP DWC), while adding new functionality that improves the discovery, modelling and distribution of enterprise data. This evolution has had no impact on the workload of SAP DWC customers. The update is performed automatically, so no migration is required. This will be the case for all new functionalities Accessing and using diverse data from different information systems and locations such as cloud providers, on-premise systems and different data sources has always been a complex challenge for organisations. The processing that this data undergoes in order to centralise it and return it afterwards (extraction, transformation, export, etc.) leads to the loss of its business context. SAP Datatsphere gives its customers the benefit of an enterprise data fabric architecture that harmonises critical data (SAP and non-SAP) in a single way and rapidly delivers meaningful data with business context and logic. SAP Datasphe
Read more

SAP Datasphere Data Integration - Part 1 - Introduction and Integration using Remote Tables

Image
SAP Datasphere provides a large set of default connections to access data from a wide range of sources, source from SAP as well as non-SAP sources or partner tools, residing in an cloud as well as on-premise environment. Connections are defined in a separate view which can be found in the main navigation window of the left. Connections are individual objects in SAP Datasphere. They are created and maintained per SAP Datasphere Space which means only members of a specific Space are able to make use of the related connections. These connections can be then be used within the different tools (like SQL View Builder, Graphical View Builder or Data Flow) for creating your data models. SAP Datasphere can integrate with both SAP or Non-SAP Source systems and it can be either cloud or on-premise. We have multiple connection types in SAP Datasphere. Each connection type supports a defined set of features. Depending on the connection type and the connection configuration, a connection can be used
Read more

SAP Datasphere Spaces

Image
  A space is a secure area created by an Administrator, in which members can acquire, prepare, and model data. The Administrator allocates disc and in-memory storage to the space, set its priority, and can limit how much memory and how many threads its statements can consume. If the administrator assigns one or more Space Administrators as members of the space, they can then assign other members, create connections to source systems, secure data with data access controls, and manage other aspects of the space. Spaces are virtual work environments with their own data bases. Spaces are decoupled, but are open for flexible access, thus allowing your users to collaborate without having to worry about sharing their data. To model your data and create stories, you need to start off with a space. You can decide how much and what kind of storage you need, as well as how important your space is compared to other spaces. You also add space members and set up connections here. Each spaces can be
Read more