Automated data pipeline and orchestration for ML-based Algorithmic Trading

TransCloud built the end-to-end automation of data pipelines which is vital for the ML-based algorithmic trading with the help of Refinitiv APIs and Interactive Brokers.

The startup which intends to solve some of the interesting problems around the Financial services and building a new product to help users with their ML-based algorithmic trading.

As the product is just evolving, it is critical for the client to experiment with various aspects of the architecture for the core offering. The data ingestion pipeline has to be highly robust for the algorithm to reliably work and the states have to be managed well. The data has to travel in a secured network between the components and the preferred architecture is to go with the cloud-native approach.

The client engaged TransCloud to build a system which can solve the challenges and meet the business needs. The whole purpose of the engagement is to have a cost-effective and scalable module so it makes the client onboarding smooth along with ease of management overall.

• Airflow based data ingestion pipelines which track every single step in each job every time it runs
• Orchestrated a scalable ML pipeline
• The whole solution communicates in a protected private network
• Seamless integration with Refinitiv APIs and InteractiveBrokers APIs
• Cost-effective and ease of maintenance
• Centralized logging and monitoring
• The whole pipeline works as a framework which can be extended to adopt the future changes
• Various Google Cloud services are used such as Google Cloud Storage, BigQuery, Compute Engine VM, Cloud Functions, Cloud Composer, VPC network, Cloud IAM, Logging and Monitoring along with external APIs like Interactive Broker and Refinitiv.