Job Title: Lead Data Engineer About Luupli: Luupli is a groundbreaking social media app driven by the principles of equity, diversity, and equality. Our mission is to ensure that social media serves as a force for good, creating a platform that maximizes the benefits for creators and businesses while positively impacting society and the planet. Although our app is still under development, we are excited about the potential it holds. Join our team of passionate individuals dedicated to making Luupli a success.
About the Role: We are looking for an experienced Lead Data Engineer specializing in recommendation systems to join our innovative team. Your contributions will be crucial in refining our backend systems, managing databases, and optimizing recommendation algorithms. This role offers an exciting opportunity to enhance the architecture and performance of our recommendation infrastructure, ensuring users receive accurate and relevant suggestions in real-time.
Responsibilities: Data and Database Management: - Design and optimize data structures and tables to effectively support recommendation and trending content.
- Utilize structured data storage solutions, notably PostgreSQL and JSONB, for managing recommendation and interaction data.
Recommendation Systems: - Develop and fine-tune recommendation algorithms (e.g., collaborative filtering, content-based, and hybrid approaches) to optimize relevancy.
- Leverage similarity search libraries like Annoy or Faiss to bolster recommendation speed and accuracy.
- Continuously assess and enhance recommendation logic to ensure it aligns with user preferences for real-time delivery.
Data Aggregation and Analysis: - Aggregate, analyze, and process user interaction data to inform recommendations and trending content.
- Create efficient queries and implement aggregation techniques to extract relevant data and insights.
Debugging and Optimization: - Pinpoint, troubleshoot, and rectify data handling issues to maintain accurate recommendations.
- Optimize queries, processing workflows, and containerized services for high performance and scalability within AWS ECS.
Requirements: - Demonstrated experience as a Data Engineer or Backend Engineer, particularly with recommendation systems.
- Proficient in SQL and adept in database management, specifically with PostgreSQL and JSONB.
- Strong understanding of various recommendation algorithms (collaborative filtering, content-based, and hybrid).
- Hands-on experience with similarity search libraries such as Annoy or Faiss.
- Proficient in Python, with experience developing backend logic for data-intensive applications in a container-based environment.
- Familiarity with AWS ECS for container orchestration, including task scheduling and scaling.
- Experience using AWS Event Bridge for triggering workflows and automating tasks based on application events.
- Strong analytical skills for data aggregation, querying, and insights generation.
- Exceptional debugging and optimization skills for large-scale data processing in cloud environments.
Preferred Qualifications: - Experience with data aggregation pipelines, ETL processes, and large-scale data management.
- Familiarity with additional AWS services (such as S3 and Lambda) for data storage and event-driven architectures.
- Experience with machine learning tools or libraries relevant to recommendation systems.
Compensation: This is an equity-only position, presenting a unique chance to gain a stake in a rapidly growing company and directly influence its success. 1% equity offered.