eCube GUI: Managed webapp to input data for Crawling and Parsing of website for info using python (django/flask framework). Trained and guided Team of 10 developers in python and related frameworks and services.
eCube Crawling Scripts: Made Crawler and Parser Standard Template to handle different site designs and to be able to easily customise individual site script. Cracking competitor site's aggressive blocking of Crawling attempts.
eCube Crawling System: Design and implement system to crawl 1000000 URLs per day using RabbitMQ for Queuing and python requests and selenium to scrap data. Handled upwards of 5cr data points gathered through crawling in Mongo and SQL.
API Integration with Client: Interacted with Client to understand requirement. Designed Micro-services Architecture and API Documentation to satisfy Client requirements. As Tech Lead guided developers in implementation, deployment of API and Integration with Client Side APIs.
Heavy API Development: Developed an API capable of serving 3 million records from SQL table in Single Call. Achieved Rate of 1000 records per sec. API Call doesnt need chunking or pagination to such large volume.
