Access & Import Data from Every Corner of Your Enterprise

Incorta’s data connectors can import data from a wide variety of sources including databases, file systems, application systems, query services, data lakes, and custom sources.

Enterprise-Class Data Acquisition

Incorta deftly handles large contemporaneous and historical data sets, opening up new possibilities for data analysis and innovation.

To make such large volumes of data ready for analytics at the speed of business, we had to rethink the way data is acquired and prepared.

 

Solve the Requirements of Enterprise-Scale Workloads

Data connectors are a complimentary part of Incorta’s data acquisition process that includes the data loader, schema mapping, and Direct Data Mapping™.

Connectors are responsible for extracting data from source systems and applications, providing schema information, and standardizing the data structures.

By contrast, the data loader handles scheduling, orchestration, data management, metadata management, compaction, and logging.

data-connectors-main-diagram_92c8caeab7e89baf7ed6cc15bf2819c0

The details of each data source, including connection details and data formats, are abstracted from the data loader. For example, extracting records from a SQL database involves opening a connection and executing a query. Extracting records from a CSV file involves opening the file, reading it line by line, and parsing the text. Extracting records from an application may involve executing a server-side routine.

Incorta presents the user with a configuration page for each data connector type. It is easy to add credentials, parameters, filters, and other properties as needed by the connector.

Connectors are extensible. The Incorta Connector SDK allows new data sources to be dynamically included without impacting existing data sources. With over 40 connectors available today, and an expanding ecosystem of connectors being developed by Incorta and its partners, any data can now be acquired, enriched, and analyzed with unprecedented speed and agility.

connector-image1_0caae5d23f25cce532db1268b41e24b7

PERFORMANCE, FLEXIBILITY, STANDARDIZATION, AND EXTENSIBILITY.

Incorta data connectors are primarily designed to be performant. They support full and incremental loads, parallel loads, and chunking, to maximize efficiency.

The connector design is also flexible in that, depending on the data origin, extraction logic can be optimized at the source. A connector can invoke source-side logic, such as application-level procedures, SQL joins, and API calls to return chunked or partial file transfers.

Connectors not only return data, but they also obtain metadata from the source system. Metadata is used during schema creation to discover and label the data elements to be extracted and mapped into the data lake.

Connector Types

Incorta supports an extensive array of data sources including files, application systems, and custom sources.

incorta-data-connector-862020

In addition to remote data sources, local data files can be uploaded directly into memory from a local disk. Incorta supports these file types: CSV, GZIP, JSON, TSV, TAB, TXT, XLSX, XML and ZIP files.

New connectors are being continually developed, including connectors to Google Sheets, Marketo and Splunk which are still experimental.

The Incorta Connector SDK makes it possible for customers and 3rd parties to create new connectors that run as first-class system components to expand the Incorta ecosystem as new requirements arise.

To view a list of currently supported data source connectors and their specific properties and parameters visit Supported Data Source Connectors. From there, you can choose a data source connector for more details on the connection process.