24th March 2023
Data Engineer
Engineering
London
£85k per annum
Data Engineer
Hybrid, London
85k per annum
Spencer Rose are looking for a Data Engineer to aid our client with a high-level greenfield project. Our client is a world-leading provider of market infrastructure, providing access to the global financial and commodities markets.
This role sits within the Data Science Team and you would be responsible for day-to-day technical development and delivery of data pipelines into the organisation’s data platform. To be successful in this role, you will be able to demonstrate knowledge and experience within data pipeline delivery, ETL/ELT, data management, solution design and delivery.
Data Engineer Responsibilities:
- Design and develop data engineering assets using SQL and AWS Services such as Lake Formation, Glue, Lambda, Athena, Kinesis
- Work closely with the Data & BI team to build, test and deploy ETL, reporting and analytics solutions within the Cloud Data Platform
- Ensure data integrity within reports and dashboards by reviewing data, identifying and resolving gaps and inconsistencies
- Maintain data within a data lake environment, ensuring delivery of high-quality data outputs
- Manage production of data pipelines and promote dev ops culture within data team Eg. automating manual processes, optimising data delivery, redesigning infrastructure for greater scalability, and rationalising the reporting suite to increase customer usage and reduce duplicated effort
Data Engineer Skills:
- 3+ years hands-on experience building, scheduling and monitoring scalable data pipelines within AWS cloud, using Spark, AWS Glue, Lake Formation, Python
- Python and Spark, but knowledge of C# is advantageous
- Strong understanding of complex data lineage in databases, schemas, tables, columns
- Knowledge of current Datamodelling techniques, including conceptual, logical and physical data models
- Experience using relational/SQL databases eg Microsoft SQL Server, Postgres
- Experience with different varieties of data, including json, xml, parquet, protobuf etc.
Consultant -
Telephone:
Share This Position
