Kedro basics — 5 min read

Introducing your new team lead…Kedro

This post explains how Kedro can guide an analytics team to follow best practices and avoid technical debt.

5 Apr 2023 (last updated 10 May 2023)
small square ice dunes

In a recent article, I explained that following software principles can help you create a well-ordered analytics project to share, extend and reuse in the future. In this post we’ll review how you can benefit from using Kedro as a toolbox to apply best practices to data science code.

How data science projects fail

As data scientists, we aspire to unlock valuable insights by building well-engineered prototypes that we can take forward into production. Instead, there is a tendency for us to make poor engineering decisions in the face of tight deadlines or write code of dubious quality through a lack of expertise. The result is technical debt and prototype code that is difficult to understand, maintain, extend, and fix. Projects that once looked promising fail to transition past the experimental stage into production.

“A cycle of quick and exciting research leads to high expectations of great improvement, followed by a long series of delays and disappointments where frustrating integration work fails to recreate those elusive improvements, made all the worse by the feeling of sunk costs and a need to justify the time spent.”

Joe Plattenburg, Data Scientist at Root Insurance

How to write well-engineered data science code

When you start to cut code on a prototype, you may not prioritize maintainability and consistency. Adopting a team culture and way of working to minimize technical debt can make the difference between success and failure.

Some of the most valuable techniques a data scientist can pick up are those that generations of software engineers already use, such as the following guidelines:

Use a standard and logical project structure: It is easier to understand a project, and share it with others, if you follow a standard structure.

Don’t use hardcoded values: instead, use precisely named constants and put them all into a single configuration file so you can find and update them easily.

Refactor your code: In data science terms, it often makes sense to use a Jupyter notebook for experimentation. But once your experiment is done, it’s time to clean up the code to remove elements that make it unmaintainable, and to remove accidental complexity. Refactor the code into Python functions and packages to form a pipeline that can be routinely tested to ensure repeatable behaviour.

"Testing after each change means that when I make a mistake, I only have a small change to consider in order to spot the error, which makes it far easier to find and fix."

Martin Fowler, Author of Refactoring: Improving the Design of Existing Code

Make code reusable by making it readable: Write your pipelines as a series of small functions that do just one task, with single return paths and a limited number of arguments.

Many data scientists say they’ve learned from their colleagues through pair programming, code reviews and in-house mentoring that enables them to build expertise suitable to their roles and requirements.

We see Kedro as the always-available team lead that steers the direction of the analytics project from the outset and encourages use of a well-organized folder structure, software design that supports regular testing, and a culture of writing readable, clean code.

What is Kedro?

Kedro is an open-source Python toolbox that applies software engineering principles to data science code. It makes it easier for a team to apply software engineering principles to data science code, which reduces the time spent rewriting data science experiments so that they are fit for production.

Kedro was born at QuantumBlack to solve the challenges faced regularly in data science projects and promote teamwork through standardised team workflows. It is now hosted by the LF AI & Data Foundation as an incubating project.

Kedro = Consistent project structure

Kedro is built on the learnings of Cookie Cutter Data Science. It helps you to standardise how configuration, source code, tests, documentation, and notebooks are organised with an adaptable project template. If your team needs to build with multiple projects that have similar structure, you can also create your own Cookie Cutter project templates with Kedro starters.

Kedro = Maintainable code

Kedro helps you refactor your business logic and data processing into Python modules and packages to form pipelines, so you can keep your notebooks clean and tidy. Kedro-Viz then visualises the pipelines to help you navigate .

“People started from scratch each time, the same pitfalls were experienced independently, reproducibility was time consuming and only members of the original project team really understood each codebase…

We needed to enforce consistency and software engineering best practices across our own work. Kedro gave us the super-power to move people from project to project and it was game-changing. After working with Kedro once, you can land in another project and know how the codebase is structured, where everything is and most importantly how you can help”.

Joel Schwarzmann, Principal Product Manager, QuantumBlack Labs, blog post

Kedro = Code quality

Kedro makes it easy to avoid common code smells such as hard-coded constants and magic numbers. The configuration library enables your code to be reusable through data, model, and logging configuration. An ever-expanding data catalog supports multiple formats of data access.

Kedro also makes it keep your code quality up to standard, through support for black, isort, and flake8 for code linting and formatting, pytest for testing, and Sphinx for documentation.

Kedro = Standardisation

Kedro integrates with standard data science tools, such as TensorFlow, scikit-learn, or Jupyter notebooks for experimentation, and commonly used routes to deployment such as Databricks.

When you follow established best practice, you have a better chance of success.

Software engineering principles only work if the entire team follows them. A tool like Kedro can guide you just like an experienced technical lead, making it second nature to use established best practices, and supporting a culture and set of processes based upon software engineering.

Look forward to greater collaboration and productivity with Kedro in your team!

Find out more about Kedro

There are many ways to learn more about Kedro:

On this page:

Photo of Jo Stichbury
Jo Stichbury
Technical Writer, QuantumBlack
Share post:
Mastodon logoLinkedIn logo

All blog posts

cover image alt

Kedro-Viz — 6 min read

How to share a Kedro-Viz pipeline visualisation

How to set up and use "publish and share" to host a Kedro-Viz project on Amazon S3.

Nero Okwa

20 Nov 2023

cover image alt

Kedro newsletter — 5 min read

In the pipeline: November 2023

From the latest news to upcoming events and interesting topics, “In the Pipeline” is overflowing with interesting updates for the Kedro community.

Jo Stichbury

7 Nov 2023

cover image alt

Best practices — 5 min read

Supercharge your data science notebook with Kedro

This article describes the “code smells” typical to a data science project developed in a Jupyter notebook. It explains how some of Kedro's features can improve the code within your notebook.

Amanda Koh

13 Oct 2023

cover image alt

Kedro newsletter — 5 min read

In the pipeline: October 2023

From the latest news to upcoming events and interesting topics, “In the Pipeline” is overflowing with interesting updates for the Kedro community.

Jo Stichbury

6 Oct 2023

cover image alt

Kedro newsletter — 5 min read

In the pipeline: September 2023

"In the Pipeline" is overflowing with the September's Kedro news.

Jo Stichbury

6 Sept 2023