Description:
Job Title: Lead Data Engineer - PySpark & Data Ingestion
Location: Sydney, NSW, Australia
Salary: $190,000 + Super
Industry: Financial Services
️ Type: Full-time | Hybrid Working Environment
Your Impact
We're looking for a skilled Lead Data Engineer to join our Information Security team and play a key role in delivering and supporting our Security Insights platform powered by Palantir . You'll be at the heart of a critical function-designing, implementing, and operating data pipelines that secure and strengthen our organisation's future.
What You'll Be Doing
- Transform unstructured datasets into consumable formats using PySpark/Python or Apache Spark (Java/Scala considered too).
- Ingest data from diverse sources (REST APIs, SFTP, Bash, SQL) while managing network connectivity, firewalls, and proxies in Linux environments.
- ⚙️ Design and optimise data pipelines in distributed systems, using smart techniques like pagination, partitioning, and broadcasting .
- Improve operations and CI/CD processes while working within Agile/Scrum and tools like Jira and ServiceNow .
- Act as a Tech Lead / 2IC , owning stakeholder engagement and promoting strong data governance and management practices.
What You'll Bring
- ✅ Deep hands-on experience with PySpark , Python , and distributed data systems.
- Strong background working with data ingestion pipelines and security-aware architectures.
- Confidence working in Linux environments with system-level integrations.
- Experience with ITSM , CI/CD pipelines, and Agile delivery practices.
- Passion for data quality , ownership , and stakeholder engagement.
Nice to Have (But Not a Dealbreaker!)
- ️ Experience with data governance , access control , and master data management .
- Familiarity with multi-tenant data lake platforms like Databricks , Snowflake , or Cloudera .
Why You'll Love It Here ❤️
- ️ Be part of a 200+ year legacy, focused on building better futures.
- Grow your career in a secure, innovative, and supportive environment.
- Enjoy flexible working , paid parental leave , and ongoing super contributions during extended leave.
- Join a purpose-driven team with a strong can-do culture and endless learning opportunities.
Ready to turn big data into big impact? Apply now and help shape the future of secure, data-driven decision-making!