Data Pipes are queues of information with specific business purposes. That information is moved in and out of Data Pipes by message brokers. An example of such a message broker is Azure Service Bus.
From a high level view, Data Pipe architecture integrates systems without requiring them to know much about one another. Typically large corporations deploy them to move a large amount of data across systems and drive ingestion processes used for analytics and reporting.
Consider a typical architecture that integrates two systems. From the 40K foot level, one system pushes data to a storage area where the other system picks it up. Both systems need to know a lot about one another. Also, compliance and reporting data need to be maintained equally on both systems. Likewise change management and data governance impact both systems. With this architecture, maintenance requires dedicated IT staff, a lot of documentation, an understanding of where risks to the business are hiding and a data analytics team. Generally, reoccuring costs are generated from human resources.
Now consider integrating the same two systems using Data Pipes. The Data Pipe platform is its own system architected to specific compliance, security and business needs. Systems connecting to it do so under rigid guidelines that indirectly enforce compliance, security and data governance rules. These platforms generally will not see significant change over time, so reoccuring costs come from data flow mostly.
Lets run some numbers and see how a Data Pipe architecture compares to a traditional integration architecture. Estimates are take from past projects for an average SMB and may not reflect current costs.
Tradition Integration Costs
Systems 1 : Data resource to map fields at $200/hour for 8 hours and a total of $1600. A Developer will need to serialize the data so it can move over the internet. This adds another $125/hour for about 4 hours and another $500. Add another 3 hours of testing $200/hour for another $600 dollars.
Systems 2: The cost will be near the same as System 1 with the addition of an ETL process needed to convert the data from System 1 into something usefull to Systems 2. That process will call on the data team once again for an additional $200/hours for about 4 hours adding another $800.
The traditional architecture used to integrate two systems costs about $6200 up front and a run rate of about 1 hour in monitoring and bug fixes for about $200/month run rate.
Data Pipes Integration Costs
System 1: For the most part it will be the same as the traditional architecture, $2700.
System 2: For the most part it will be the same as the traditional architecture, $3500.
There is the setup cost of the data pipe. This will be a cloud architect at $150/hours for about 8 hours and a total of $1200. The cost of the data pipe platform has a number of cost items to keep in mind. In the case of Azure Message Service the cost of the platform runs at $0.0135/hour, each Data Pipe will be billed at $0.80/million calls to it. A million calls in one month for a SMB is a lot! There are connection charges at $.03/connection. Reflecting back over the years I will average the number of calls to a business with 6 applications half of which are integrated and passing data twice a day. So the total cost of a Data Pipe is a upfront setup at $7400 dollars and a run rate of around $16/month.
In summary, traditional integration of two systems cost about $6200 up front and $200/month, while using data pipes cost around $7400 up front and $16/month. Note that costs will change when considering reserve pricing and auto scaling. Which architecture is best for your business depends on your compliance, security, business volume fluctuations and accounting practices. GOGI can help you make that determination, CONTACT US.
