Senior Data Engineer

The Data & Insight team is part of Digital, Data & Customer Services.

This team is a central data and insight function made up of data analysts, data engineers and cloud engineers supported by a product manager and delivery manager. Our vision is to help Hackney maximise the value of its data so that the council can make better decisions, do more with less, and proactively support our residents.

Recently we have completed some great work including:

– Building one of local government’s most advanced cloud data analytics platforms

– Creating our own reusable web map template to share spatial information with our residents

– Building machine learning models to match resident data, conduct predictive analysis and forecast trends

We are a collaborative, curious and friendly team and are now looking to grow.

The Data & Insight team supports services across the council, from parking to adult social care, to get value from the vast amount of data we collect and promote best practice for data analysis.

We do this by:

– Helping colleagues access trusted data

– Providing simple, secure and reusable tools to enable analysts’ best work

– Supporting users and building analytical capabilities

– Innovating with new technologies and conducting cross-cutting analysis

– Leading the data analyst community of practice and the strategic use of data in the council

As a senior data engineer, you will lead on the design and use of Hackney’s cloud data analytics platform in AWS to provide efficient and reliable data pipelines.

To do this you will:

– Collaborate with a senior data architect, cloud engineers, software engineers, data analysts and other data engineers

– Select and implement appropriate technologies to build, orchestrate and monitor data pipelines

– Develop reusable data ingestion methods for a range of data sources

– Set standards and create high quality, reusable code blocks for efficient data transformation

– Design and implement a warehouse layer that meets the needs of data analysts and consumers

– Productionise machine learning models

– Mentor and develop others to build capability within the team and the data analyst community

We don’t expect you to tick every box on our job description. The most important things to us are that you: 

– Want to deliver data products that are a force for good

– Have a track record of delivering efficient, scalable and automated data pipelines

– Have experience designing and implementing data solutions in AWS

– Are skilled in using Python, SQL and Apache Spark

– Enjoy supporting others to learn, whilst continuing to develop your own skills

– Can work in a fast-paced environment, as part of a diverse team

Over the next year we plan to:

– Make ingestions into the data lake more reusable and cost effective so that we can serve more data through our platform

– Redesign the warehouse layer of the platform to reliably serve data in a range of formats to users and applications

– Further integrate data governance processes with access controls to enable easier data sharing between council services

– Collaborate with the Customer Experience team and technical leads to deliver joined up resident data for front-line staff

To do this we need a brilliant team, with the right people and skills in place.

We have strong staff retention in our team because many of us love Hackney so much that we can’t imagine working anywhere else. We love: 

– Solving real world problems to improve the lives of residents

– That our views and projects are valued by our leadership

– Being able to work flexibly

– Working alongside a diverse range of colleagues

– That we’re encouraged to learn and given the space to do it

– The generous annual leave and unrivalled pension scheme

We work remotely most of the time. We all gather in the office as a team at least once a fortnight as a team and together with the wider data analyst community of practice, however many of us prefer to work from the office more frequently.

If you’d like an informal chat about the role, you can speak to Jo Cumper, Strategic Lead for Talent Acquisition for Digital & Data. Book an appointment with Jo or email joanne.cumper@hackney.gov.uk.

There is no better place to work in London than Hackney; we have great pride in our borough, our communities, and our services. Hackney is one of London’s most vibrant and diverse boroughs. It’s one of the most sought after areas to live in London with good schools, parks and local amenities, as well as great transport links and vibrant and diverse communities.

you want to work in a place where you can represent our values so we achieve the best for our residents, please click on the apply button below.

The recruitment process is anonymous. Therefore, we do not accept supporting statements or CVs.

When applying, there will be a number of questions relating to the post on the application form. It is essential that you complete the application questions, responding to each using examples of your skills, knowledge and experience.

We will not accept applications where Artificial Intelligence (AI) has been used to assist with completing the competency questions or any part of the application form.

This application process replaces a supporting statement.

Closing date for applications: 08 October 2023 (22:59 pm)

Interview date: TBC

We are also working towards our vision, which is to be a place for everyone; where residents and staff can be proud; a place that celebrates diversity and where everyone can feel valued, included and involved. In order to achieve this, we look for people who are: Proud; Ambitious; Pioneering; Open; Proactive; Inclusive.

Hackney Council works to eradicate discrimination on the basis of age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex, and sexual orientation. We also recognise that people can be disadvantaged by their social and economic circumstances, so we will also work to eliminate discrimination and disadvantage caused by social class. We also welcome those interested in flexible working. We particularly welcome applications from disabled people, as this group is currently under represented in our workforce.

The Council has a dedicated employment support service (Hackney Works), which offers Hackney residents guidance in relation to applying for jobs. If you are thinking about applying for a job, please contact the Hackney Works team to find out what support is available: hackneyworks@hackney.gov.uk