Medallion Architecture with Incorta Explained

‍

As organizations accumulate massive volumes of data across CRMs, ERPs, and countless other systems, the ability to effectively manage and extract meaningful insights from this information has become a strategic imperative.The Medallion architecture, championed by Databricks, offers a structured approach to this challenge by organizing data into distinct layers: Bronze, Silver, and Gold. But how does Incorta, with its unique capabilities, fit into this powerful framework? Let's explore.

The Medallion architecture is designed to progressively refine data, ensuring quality, and provide a clean, business-ready view for analysis. Incorta's architecture, with its focus on speed, Direct Data Mapping®, and schema-on-query, complements this approach beautifully.

‍

‍

The Bronze Layer: Your Raw Data Foundation

The Bronze layer is your data's untouched sanctuary. It's where raw, unvalidated data lands in the Incorta Data Lake, preserving its original format. Think of it as a historical record, capturing every change (including updates and deletions) through Change Data Capture (CDC) logic using our connectors. Your data lake can also be integrated into Incorta through remote tables, giving you unified access to all your data sources.

‍

What you can achieve in Incorta’s Bronze Layer: 

• Seamless Ingestion: Incorta's robust data connectors make it incredibly easy to bring in data from diverse sources.
• Efficient Loading: Support for incremental data loads means you're only processing new or changed data, saving time and resources.
• Automated Refreshes: Schedule refreshes to keep your Bronze layer up-to-date without manual intervention.
• Data Retention Management: Incorta can help you purge old data, ensuring your Bronze layer remains manageable.

The Silver Layer: Cleaning, Validating, and Staging Your Data

The Silver layer is where the magic of data preparation begins. This layer is source-aligned, meaning it's still close to the original data but undergoes crucial validation and transformation. It's your staging area, where data quality is prioritized.

What You Can Achieve in Incorta's Silver Layer:

• Schema Organization: Create separate schemas with table aliases to logically group your data for better management.
• Data Cleansing with Formula Columns: Leverage Incorta's powerful formula columns to clean, standardize, and enrich your data.
• Partitioning datasets into tailored subsets for specific groups or departments to support downstream analysis
• Incorta Analyzer Tables: Utilize Analyzer Tables for data validation and initial aggregations.
• Cost-Effective Transformations with Spark-based MVs: For heavy-duty transformations, utilize Spark-based Materialized Views within Incorta, offering a cost-effective solution. While not always necessary, MVs can be employed for complex pre-calculations if required.
• Building "Bridge Tables": Incorta's flexibility makes it ideal for creating "bridge tables" to handle many-to-many relationships, crucial for robust data models.
• Automation with Load Plans and Schedulers: Automate your data processing workflows using Incorta's load plans and schedulers.
• Data Quality with DataStudio: Leverage DataStudio for comprehensive data quality checks and monitoring.

The Gold Layer: Business-Ready Insights

The Gold layer is your business-aligned layer, meticulously modeled to serve specific business needs. Think of it as your "Business Views" or “AI-ready data” for critical processes like Order-to-Cash, Procure-to-Pay, or AI use cases.

Incorta Offers Three Powerful Options for Gold Layer Modeling:

1. Direct Modeling in a Business Schema: This is a common approach where you create separate Incorta business schemas for each subject area (e.g., "Financials") in our Semantic layer.
   
   • Aliases and Joins: Define the Incorta Metadata on your physical schema which has the source tables, and set up joins and aliases
   • Business Views: Define powerful Business Views to expose data in a user-friendly, business-centric manner in our Semantic layer.
   • Spark Views: Optionally leverage Spark Views for complex business logic and aggregations.
   • SQLx Connectivity: Query your Gold Layer using SQLx from other BI tools, enabling broad accessibility.
2. Star Schema Modeling within Incorta: For traditional data warehousing paradigms, you can model a Star Schema directly within Incorta.
   
   • Model it logically via Business Views: Model the star schema logically as business views within the Semantic Layer. These views can be built directly on Incorta’s physical schema, which stores tables in a 3NF structure. By leveraging Incorta’s Direct Data Mapping® technology, you achieve high-performance queries without the extensive ETL transformations typically required to create a physical star schema.

• Dedicated Physical Schemas: Optionally, each star schema can also have its own physical schema, offering clear organization.
• Aliases and Joins: Effectively use aliases and set up joins to create your dimensional and fact tables.
• Permission Management: Incorta's robust security allows for granular permission sharing.

3. External Star Schema Storage: If your use case demands it, you can push your refined star schema data to external destinations like Google BigQuery, Snowflake, or Databricks.

The Synergistic Power of Incorta and Medallion

By integrating Incorta into the Medallion architecture, organizations can create a powerful and efficient data pipeline. Incorta's ability to directly ingest and rapidly query large datasets, combined with its flexible modeling capabilities, accelerates data transformation from raw to refined.

From the raw intake of the Bronze layer, through the meticulous cleaning and staging of the Silver layer, to the business-ready insights of the Gold layer, Incorta empowers you to build a true data powerhouse. This approach not only ensures data quality and governance but also provides the agility needed to respond quickly to evolving business demands, transforming your data into a strategic asset.

‍