June 2025 – August 2025
Software Engineer Intern
DatalynnNew York, NY
- Developed a scalable and modular Python web-crawling system with on-demand data updates and deployed it as a reusable plugin in staging and production environments.
- Built high-concurrency scraping pipelines using Playwright, asyncio, and aiohttp, reducing end-to-end latency by 60% and processing more than 180,000 rows per day.
- Processed collected data with Pandas, stored structured results in MySQL, and exposed retrieval, update, and deletion operations through FastAPI.
- Led requirements analysis, system design, and implementation of core components, improving overall system efficiency by 4× compared with the initial design.
- Enabled distributed deployment across five workers, increasing horizontal processing capacity by 5× and improving crawling throughput from 0.5 GB/hour to 2 GB/hour.
- Added structured logging, error handling, and recovery mechanisms to improve stability and maintainability.
- Python
- FastAPI
- Playwright
- asyncio
- aiohttp
- MySQL
- Pandas