Job Objective / Purpose
Responsible to design, build, and optimize systems for data collection, storage, access, and analytics at scale. They create data pipelines used by data analyst, data-centric applications, and other data consumers.
This IT role requires a significant set of technical skills, including deep knowledge of SQL database design and multiple programming languages. Data engineers also need communication skills to work across departments and to understand what business leaders want to gain from the company’s large datasets.
Data engineers are responsible for building algorithms for accessing raw data, but to do this, they need to understand a company’s or client’s objectives, as aligning data strategies with business goals is important, especially when large and complex datasets and databases are involved.
Data engineers must demonstrate outstanding interpersonal skills to effectively collaborate with both data owners, report generators and analysts.
Duties and Responsibilities
- Work closely with our data analyst team to help build complex algorithms that provide unique insights into our data
- Apply secure by design to ensure data confidentiality, integrity and availability
- Use agile software development processes to iteratively make improvements to our back end systems
- Model front end and back end data sources to help draw a more comprehensive picture of user flows throughout our system and enable powerful data analysis
- Build, test and optimize data pipelines that clean, transform, and aggregate data from disparate sources
- Develop models that can be used to make predictions and answer questions for the overall business
- Ensure compliance with data governance and security policies
Background and Qualifications
- Bachelor’s degree in Information Technology, Computer Science, Engineering or related field
- Proficiency and 2 plus years experience required on the following items
Data engineering and cloud data processing services (Google Cloud Platform or AWS/Azure - Knowledge of SQL, Java and Python, other languages
- Good understanding of ETL tools like Glue, Data Factory, and data modeling
- Open mindset, ability to quickly adapt new technologies and learn new practice
- An excellent communicator and good personal skills
- Internal/external advisory/consulting experience
- Demonstrated client management and relationship skills
- Designing and implementing data pipelines on Google Cloud or AWS/Azure
- Knowledge of at least one of the following GCP services or similar services on AWS/Azure:
- Compute Engine
- AppEngine
- Kubernetes Engine
- BigQuery
- FireStore
- CloudSQL
- PubSub
- Anthos
- Dialogflow
- Document AI
- Vision AI
- VPC Network
- IAM