Automated Data Pipeline and Orchestration for ML-Based Algorithmic Trading
Overview
Transcloud spearheaded the development of an end-to-end automated data pipeline, pivotal for ML-based algorithmic trading, leveraging third-party APIs such as Refinitiv APIs, Interactive Brokers, and custom-built Machine Learning models. This initiative aimed to streamline financial services operations.
Industry : Fin-Tech
Services
- Data Engineering
- MLOps
- Cloud-native
Challenge
A startup ventured into addressing complex challenges within the financial services sector by crafting a new product for ML-based algorithmic trading. With the product at its inception, experimentation with diverse architectural aspects was critical to ensuring robustness in data ingestion and management, emphasizing a cloud-native approach.
Goal
The client engaged Transcloud to tackle challenges and align with business objectives. The engagement aimed to deliver a cost-effective and scalable solution, facilitating seamless client onboarding and overall management.
Technical Excellence
- Implemented Airflow-based data ingestion pipelines to meticulously track every step in job execution.
- Orchestrated a scalable ML pipeline to efficiently process and analyze data.
- Ensured secure communication within a protected private network.
- Achieved seamless integration with Refinitiv APIs and Interactive Brokers APIs.
- Prioritized cost-effectiveness and ease of maintenance.
- Implemented centralized logging and monitoring for enhanced visibility.
- Designed the pipeline as a flexible framework capable of accommodating future changes.
Key Technologies Used
- Google Cloud Storage
- BigQuery
- Compute Engine VM
- Cloud Functions
- Cloud Composer
External APIs
- Refinitiv
- Interactive Brokers
Conclusion
The collaboration between Transcloud and the confidential client exemplifies the significance of strategic partnerships in tackling complex challenges within the financial services domain. By leveraging cutting-edge technologies and adopting a cloud-native approach, the project achieved significant milestones in streamlining data processing, enhancing scalability, and ensuring cost-effectiveness. The seamless integration with external APIs and meticulous attention to technical excellence underscored Transcloud’s commitment to delivering innovative solutions tailored to meet evolving business needs.
