Data connectors are software components that automatically extract data, sometimes periodically, from one or more upstream data sources and land that data in another database. Data connectors are common in most business intelligence, analytics and data science applications and frameworks.
As business transactions have increasingly shifted online, most organizations collect vast amounts of data. This information resides in multiple locations, including traditional databases, file systems, key-value stores, object stores and streaming data sources.
Types of data sources
While organizations once stored most of their information in on-premises databases, with the rise of cloud computing, information can be spread far and wide. Today, an organization’s data may reside in cloud objects stores, software-as-a-service (SaaS) platforms such as NetSuite or Salesforce, cloud-resident data warehouses, and even tools such as Google Sheets. In many organizations data can also reside in hundreds or thousands of spreadsheets, which are typically siloed and disparate sources. The more varied the information sources, the richer the analytics and decision quality, so it is imperative that organizations be able to connect to data wherever it resides.
Data connectors do more than just transfer data
To be effective, data connectors need to do more than just access and transfer data from remote data sources. Ideally, a data loading service should handle details such as managing credentials and endpoints, applying filters, and accommodating parameters specific to each datastore to securely and transparently ingest data. Data connectors also need to handle full or incremental loads either on demand or according to preset schedules. Another important capability is support for “upserts” — updating records when they already exist and inserting new records when they do not.
Data connectors are frequently highly optimized, supporting features such as parallel loads, encryption at ingest, webhook callbacks and mechanisms to access data sources via proxies across firewalls. Without these capabilities, data connectors would be of less value in real-world applications.
Incorta data connectors
Incorta’s Direct Data Platform™ makes it easy to connect to any data source in the enterprise. With Incorta, organizations can realize a modern, durable data hub for multiple analytic and data science applications. With over 200 data connectors and additional connectors being added all the time, Incorta can access data wherever it resides. Analysts and data scientists benefit from timely, high-performance access to all data sources for better, faster decisions and improved organizational agility.