Incorta deftly handles large contemporaneous and historical data sets, opening up new possibilities for data analysis and innovation.
To make such large volumes of data ready for analytics at the speed of business, we had to rethink the way data is acquired and prepared.
Data connectors are a complimentary part of Incorta’s data acquisition process that includes the data loader, schema mapping, and Direct Data Mapping™.
Connectors are responsible for extracting data from source systems and applications, providing schema information, and standardizing the data structures.
By contrast, the data loader handles scheduling, orchestration, data management, metadata management, compaction, and logging.
The details of each data source, including connection details and data formats, are abstracted from the data loader. For example, extracting records from a SQL database involves opening a connection and executing a query. Extracting records from a CSV file involves opening the file, reading it line by line, and parsing the text. Extracting records from an application may involve executing a server-side routine.
Incorta presents the user with a configuration page for each data connector type. It is easy to add credentials, parameters, filters, and other properties as needed by the connector.
Connectors are extensible. The Incorta Connector SDK allows new data sources to be dynamically included without impacting existing data sources. With over 40 connectors available today, and an expanding ecosystem of connectors being developed by Incorta and its partners, any data can now be acquired, enriched, and analyzed with unprecedented speed and agility.
PERFORMANCE, FLEXIBILITY, STANDARDIZATION, AND EXTENSIBILITY.
Incorta data connectors are primarily designed to be performant. They support full and incremental loads, parallel loads, and chunking, to maximize efficiency.
The connector design is also flexible in that, depending on the data origin, extraction logic can be optimized at the source. A connector can invoke source-side logic, such as application-level procedures, SQL joins, and API calls to return chunked or partial file transfers.
Connectors not only return data, but they also obtain metadata from the source system. Metadata is used during schema creation to discover and label the data elements to be extracted and mapped into the data lake.
Incorta supports an extensive array of data sources including files, application systems, and custom sources.
In addition to remote data sources, local data files can be uploaded directly into memory from a local disk. Incorta supports these file types: CSV, GZIP, JSON, TSV, TAB, TXT, XLSX, XML and ZIP files.
New connectors are being continually developed, including connectors to Google Sheets, Marketo and Splunk which are still experimental.
The Incorta Connector SDK makes it possible for customers and 3rd parties to create new connectors that run as first-class system components to expand the Incorta ecosystem as new requirements arise.
To view a list of currently supported data source connectors and their specific properties and parameters visit Supported Data Source Connectors. From there, you can choose a data source connector for more details on the connection process.