Automated Data Pipeline and Orchestration for ML-Based Algorithmic Trading

Overview
Transcloud spearheaded the development of an end-to-end automated data pipeline pivotal for ML-based algorithmic trading, leveraging third party APIs such as Refinitiv APIs, Interactive Brokers and custom-build Machine Learning models. This initiative aimed to streamline financial services operations.

Industry: Fin-Tech

Services: Data Engineering, MLOps, Cloud-native

Challenge
A startup ventured into addressing complex challenges within the financial services sector by crafting a new product for ML-based algorithmic trading. With the product at its inception, experimentation with diverse architectural aspects was critical to ensuring robustness in data ingestion and management, emphasizing a cloud-native approach.

Goal
The client engaged Transcloud to tackle challenges and align with business objectives. The engagement aimed to deliver a cost-effective and scalable solution, facilitating seamless client onboarding and overall management.

Technical Excellence
Implemented Airflow-based data ingestion pipelines to meticulously track every step in job execution.
Orchestrated a scalable ML pipeline to efficiently process and analyze data.
Ensured secure communication within a protected private network.
Achieved seamless integration with Refinitiv APIs and Interactive Brokers APIs.
Prioritized cost-effectiveness and ease of maintenance.
Implemented centralized logging and monitoring for enhanced visibility.
Designed the pipeline as a flexible framework capable of accommodating future changes.

Key Technologies Used
Google Cloud Storage, BigQuery, Compute Engine VM, Cloud Functions, Cloud Composer

External APIs
Refinitiv
Interactive Brokers

Conclusion
The collaboration between Transcloud and the confidential client exemplifies the significance of strategic partnerships in tackling complex challenges within the financial services domain. By leveraging cutting-edge technologies and adopting a cloud-native approach, the project achieved significant milestones in streamlining data processing, enhancing scalability, and ensuring cost-effectiveness. The seamless integration with external APIs and meticulous attention to technical excellence underscored Transcloud’s commitment to delivering innovative solutions tailored to meet evolving business needs.