Apache NiFi Lead [Hands-on + Technical Oversight]

Role Summary

The Apache NiFi Lead will own the design, development, and operation of scalable data flow solutions using Apache NiFi. This role combines hands-on engineering with technical leadership, guiding developers, partnering with data, backend, and DevOps teams, and ensuring reliable, secure, and performant data pipelines across environments.
Hours: 3 PM IST to 11:30 PM IST

Key Responsibilities

Technical Leadership & Architecture

• Lead the design and architecture of NiFi-based data ingestion, transformation, and routing pipelines.
• Define standards for flow design, naming conventions, versioning, and documentation.
• Review flows and configurations to ensure scalability, performance, and maintainability.
• Mentor NiFi developers and provide technical guidance.

Development & Integration

• Build and maintain NiFi flows for batch and near–real-time data processing.
• Integrate NiFi with databases, APIs, file systems, message queues, and cloud services.
• Implement processors, controller services, and parameter contexts.
• Collaborate with backend and data teams to align data contracts and schemas.

Operations, Security & Reliability

• Configure NiFi clustering, load balancing, and high availability.
• Implement security best practices (TLS, user authentication/authorization, data provenance).
• Monitor performance, troubleshoot issues, and optimize flow efficiency.
• Support deployments, upgrades, and environment promotion (Dev → QA → Prod).

Agile Delivery & Collaboration

• Participate in sprint planning, stand-ups, reviews, and retrospectives.
• Work closely with QA and DevOps teams to ensure testability and release readiness.
• Coordinate with US-based stakeholders on requirements, priorities, and delivery timelines.

Required Skills & Experience

Technical

• 6+ years of experience in data engineering or integration roles.
• Strong hands-on experience with Apache NiFi (flow design, processors, controller services).
• Experience designing high-volume, fault-tolerant data pipelines.
• Proficiency integrating NiFi with REST APIs, databases (SQL/NoSQL), files, and messaging systems.
• Experience with NiFi security, clustering, and performance tuning.
• Working knowledge of Linux, scripting, and Git-based version control.

Data & Integration

• Understanding of data formats (JSON, XML, CSV, Avro, Parquet).
• Experience with data validation, transformation, and error handling patterns.
• Familiarity with schema management and data quality checks.

Soft Skills

• Strong leadership and mentoring skills.
• Clear communication and stakeholder management abilities.
• Comfortable working with offshore teams and US stakeholders.
• Experience in Agile / Scrum delivery environments.
• Ownership mindset with focus on reliability and delivery.

Nice to Have

• Experience with custom NiFi processors (Java).
• Exposure to cloud platforms (Azure / AWS / GCP).
• Experience with CI/CD pipelines for NiFi deployments.
• Familiarity with Kafka or other streaming platforms.
• Experience supporting production data platforms.