I was recently in a call over the weekend with a contact at RBC Capital Markets in the global front-office division. On top of my own market observations related to Reconfigured.io / others and attempts to slice-and-dice the DAG problem, the simplest approach to perennial topics seems to be related to specific end-points that can lead to value creation for shareholders through a path to frictionless data solutions, as well as optimizing organizational success through the effective valuation of internal data assets.

While public sources have postured a best-case commercial multiple on a 10-50x opportunity, defensive value analysis has been postured as a way of enhancing effectiveness for solution designs whose values are often multivariate across global markets. Considering a JetBrains survey from 2015, I have also provided a fairly one-to-one pathway on an ideal world situation with ETL / ELT across a firm within Capital Markets, leading to a solution design that can drive upwards $10-50MM in post-money with the right partners and just a subset of production data for an offering.

Excel’s versatility and the wide array of virtual money functions it offers make it an indispensable tool for scenario modeling; it is highly unlikely to go away within this vertical or across the world of money. From analyzing historical performance to predicting future outcomes, Excel provides the tools necessary to make informed decisions. Advanced functions like PMT, NPV, and IRR, combined with the ability to organize data effectively and apply consistent formatting, enable finance teams to create accurate and reliable financial models. This brings us down to the basic porting conversation within these firms’ data management practices and upwards 150 MiB data sizes; which may scale up to Exabytes within an organization of 10,000 or more also handling a CRM instance.

Using dbt, within a microservices architecture, where each service has its own database, dbt can play one stack-agnostic yet crucial role in managing data transformation and integration. By ingesting event streams from each microservice into a centralized data warehouse, dbt ensures that each service can change its database schema without affecting others. This approach not only addresses the immediate need for handling large volumes of data but also lays the foundation for scalability and adaptability as the firm grows and evolves. I have discoursed with Phind to retrieve simulated examples of what a case looks like in reality, with data in disparate sources such as CSV also being in a serialized format within other cloud stores. Replace G-Drive with the CRM view your organization has provided you with and you’ll get the idea.

We begin with a phased approach for transforming event streams into tables, let’s consider a simplified representation of a schema and logical components. This example will focus on how dbt can be used to manage data transformation and integration in this environment, addressing the challenge of handling data from multiple sources without direct database access. This is often an integral change control in place by a siloed organizational approach where business owners influence a soft control over company assets with master keys a la the FBI and others. Within distributed microservices, this side of the equation in 2024 has largely disappeared.

Practical Architecture

• Order Service: Manages order data related to lot sizes, liquidity, acquisitions, and serialized products; including order details, customer information, and payment status.
• Customer Service: Stores customer profiles, preferences, and interaction history. Native data-points within modern CRM tools.
• Salesforce Integration: Connects to Salesforce for CRM data, including leads, opportunities, and customer interactions. Native to dbt and available as a data importer within the platform.

dbt Solution Design

Step 1: Data Ingestion

• Event Streams: Use dbt macros to ingest event streams from each microservice into a centralized data warehouse, such as Snowflake. This approach ensures that each service can change its database schema without affecting others.
• Data Transformation: Transform raw event data into structured tables using dbt models. This involves cleaning, aggregating, and joining data from different sources to create a unified view of the business data.

Step 2: Data Validation

• Schema Tests: Implement dbt schema tests to validate the structure and data types of the transformed tables. This ensures data integrity and consistency across the data warehouse.
• Custom Data Tests: Use custom data tests to validate business logic and assumptions. For example, testing that the FPV matches the complex business logic of the Excel model.

Step 3: Data Integration

• Salesforce Integration: Use dbt to integrate Salesforce data into the data warehouse. This involves mapping Salesforce fields to the corresponding fields in the data warehouse schema and ensuring that data types and formats are consistent.
• Data Enrichment: Enrich data from the microservices with additional context from Salesforce, such as customer profiles and sales opportunities, to provide a more comprehensive view of the business data.

Step 4: Data Analysis and Reporting

• Dashboards and Reports: Use dbt to create dashboards and reports that leverage the integrated and enriched data. This includes creating visualizations and insights that support decision-making across the organization.
• Performance Monitoring: Implement dbt models to monitor the performance of the microservices and the data integration process. This includes tracking data freshness, transformation errors, and system performance metrics.
• Commence Time-Savings: Free up a block of those 16-hours as a type of time friction for further client or partner interaction.

So before we book a call or if you found that this has helped you with understanding the severely under-saturated yet asymmetric opportunity within the data asset management landscape, please ask yourself if your organization is equipping your contractors for success. Annual software shrink according to people like Alexei Alexis are telling. All the best …