Data Sources

Learn about the process of extracting and contextualizing data

  2 minute read  

The data connectivity feature of the Management Console consists of two parts, setup/management of connections and extracting/contextualizing data. This page explains how the second part works in the Management Console actually.

Overview

In order to retrieve data from the source, created connections shall be initialized. The initialization and deployment consist of the following steps:

  1. Check the status of the selected connection.
  2. Send a request for the data source deployment with information (nodes and topics).
  3. Retrieve the status of data sources periodically.

How the test for connections works

Although the setup process tests the reachability of the server, the status is not the latest at the initialization. Therefore, the Management Console checks the status before this initialization. You find under Connections section how this step works.

How data sources are deployed

bild
bild

If the connection status is good, a technician enters some information, such as nodes and topics, in the next step. Then, the Management Console sends the deployment request with these values. The deployment process consists of the following steps:

  1. The Management Console encrypts the request message with the payload and sends it to Kafka.
  2. The Management Console Companion receives the message and resolves its message type.
  3. The Companion executes the command helm upgrade with some parameters like the corresponding release name, the namespace, etc; that is, it modifies the values of the helm Chart to add new data sources. These values are mapped to the configMap.
  4. The Management Console Datasource Deployer checks changes in the configMap periodically.
  5. If there are changes in the configMap, it deploys data sources in the cluster with the UMH instance.

How the Management Console retrieves the status of data sources

bild
bild

The Management Console retrieves the status of data sources periodically as follows:

  1. The Management Console encrypts the request message and sends it to Kafka.
  2. The Management Console Companion receives the message and resolves its message type.
  3. The Companion filters benthos pods by using the label selector and gets the pods’ status
  4. After arriving result, it generates a message and sends a reply.
  5. The Management Console receives statuses.