Senior Data Engineer (Pyspark and Databricks)

Apply Now

Direction:

Data Engineering

Location:

EU

Position:

Freelancer

WE ARE

Opinov8 is a technology service provider, with an Opinion in Innovation, we engage with our clients at any stage in their product engineering and innovation goals, with solutions being mainly collaborative and outcome driven. We are driven by delivering value, for our clients, through technological innovation.

We are a young energetic and dynamically agile company and the team behind Opinov8 are software industry veterans. We are ambitious on our own and our client’s behalf and we do not look at a client engagement as a project, we are in it to help our clients build sustainable products, platforms, and businesses.

Our core values and what we look for in our Opinov8rs:
- You are Collaborative - is a key component to being successful in distributed environments.
- You are Innovative - it is a requirement that we innovate, as individuals and as a company.
- Be Adventurous - courageous people are adventurous, we are hiring and nurturing this quality.
- Always Respectful - remember if you don’t respect others they will not respect you.
- Inherently Intelligent - it goes without saying we are all intelligent in our own way.
- Ultimately Responsible - being responsible is a core personality trait that is needed for us and our clients’ success and ultimately a happy work environment.

PROJECT DESCRIPTION
 
Our client’s product allows you to consolidate online, telephone, and in-person payments into one user-friendly system. Their API enables seamless integration of their payment capture engine into your current platform, making settlement and payment reconciliation effortless with their powerful settlement and reporting engine. Additionally, the product offers robust transaction verification and risk scoring, catering to the needs of large financial institutions. 


IN THIS ROLE, YOU WILL 

- Play a key role in architectural decisions as we build toward an efficient innovative analytical solutions using Big Data related technologies;
- Develop and launch new features to adapt to evolving business needs;
- Develop and integrate Pyspark pipelines for data ingestion, aggregation anonymization and display;
- Deploy a range of cloud-based technologies and services for critical projects;
- Write, test, instrument, and deploy code to our Databricks + Docker via Terraform;
- Actively participate in code review and test solutions to ensure it meets best practice specifications. Maintain high standards of code quality, and encourage the same by providing constructive code reviews to collaborators;
- Troubleshoot and resolve issues, problems, and errors encountered across various systems;
- Communicate with key technical stakeholders and third parties to understand requirements and help drive decisions, and gather requirements when under specified;
- Help drive the innovation and evolution of the company products.


IF YOU ARE

- 7+ years of experience in software development, including at least 3 years of experience with writing in Python in containerized production environment;
- Ability to write high-performance, well-designed, well-crafted, readable, and tested code in a team environment;
- Good understanding of distributed computing principles, CI/CD principles and best practices;
- Solid experience with Pyspark, Data Streaming, Delta Tables, Azure Databricks (Auto Loader, Unity Catalog, Workflows, Dashboards, Asset Bundles), that includes streaming processing to joining and aggregating huge datasets, to integrating with third party data sources;
- Experience with debugging and optimizing Cloud-based data pipeline is must;
- Strong knowledge of SQL required;
- Solid understanding of both non-relational and relational data base systems (Cosmos DB, DynamoDB, MongoDB, Cassandra) and familiarity with database fundamentals, such as ACID, database schema, normalized/ denormalized data;
- Familiarity with the following Infrastructure technologies - Docker/Kubernetes/Yarn, AzureDevOps, Terraform.


AS AN OPINOV8R, YOU WILL HAVE 
 
- Digital-First Approach: Great talent knows no borders! You can work from wherever you are. We hire and collaborate with professionals worldwide.  
- Remote Work Model: Balance your professional and personal life with our flexible working conditions. We offer an, empowering you to be productive and deliver your best from anywhere.   
- Exciting Projects: Dive into impactful projects across various industries that challenge and spark your creativity. Work on innovative solutions that push boundaries, giving you the chance to make a difference and drive change. 
- Boost Your Expertise: Elevate your career with unmatched experiences that expand your knowledge and master skills. We foster continuous learning and development opportunities that allow you to grow and reach new heights. 
- Join the Best Team Ever: Collaborate with our diverse and сross-cultural team of passionate technologists and creative thinkers who inspire and support. Together, we develop services that make a real impact on the world.


HOW’S THE HIRING PROCESS GOING 

We strive to make a smooth and transparent hiring process to find the perfect match for both sides. Steps may differ depending on the position — here's what to expect: 

- Initial Interview: If your background fits the role's criteria, we'll invite you for an interview with a Talent Acquisition Specialist to get to know you better. 
- Technical Interview: Depending on the role, you may be asked to complete a technical assessment/test task to showcase your expertise.  
- Client’s Interview: Meet with client representatives to discuss the role specifics and gain insights into the project's scope and expectations.
- Final Decision: After completing all interviews and assessments, we'll reach a final decision and get back to you with the result.

INTERESTING ALREADY?

We’d love to hear from you! Apply for this position and become part of our team! Let’s make great things together!
Apply Now

Related Job Opening

Let us innov8 with you

Engineering your Digital Future through Solution Excellence Globally
UK, London
Office 9, Weyhouse, Church Street, Weybridge, KT13 8NA
Ukraine, Kyiv
BC Eurasia, 11th floor, 75, Zhylyanska Street, 01032
Egypt, Cairo
11G/4, Ahmed Kamal Street, 
New Maadi
Prepare for a quick response:
contactus@opinov8.com
© Opinov8 2021. All rights reserved.       Privacy Policy
crosschevron-down