Unit Testing Jupyter Notebooks - testbook
2021-02-20, 16:00–16:25, Prerecorded Talks

Previous attempts at unit testing notebooks involved writing the tests in the notebook itself, which was faulty, difficult to read, and even more difficult to maintain. However, testbook allows for unit tests to be run against notebooks in separate test files, hence treating .ipynb files as .py files.


Traditionally Jupyter Notebook users have had an excellent experience around exploring code solutions with an interactive development environment. However, the Jupyter Notebook document that’s produced -- the .ipynb file -- was not easily testable in the same variety of coding situations as other extension formats like plain .py files. For certain situations end-to-end execution tools like papermill allow for testing the entire document as one unit, but individual unittesting of code snippets from the file was difficult or impossible to achieve without exporting to a new format and refactoring.

Notebooks in recent years have exploded in popularity, with millions of notebooks on github alone. To alleviate the pain point for testing these notebooks, we created a new library called testbook. The testbook library is a unit testing framework for testing code in Jupyter Notebooks with pytest patterns.

Previous attempts at unit testing notebooks involved writing the tests in the notebook itself, which was faulty, difficult to read, and even more difficult to maintain. However, testbook allows for unit tests to be run against notebooks in separate test files, hence treating .ipynb files as .py files.

Features of the testbook library

  • Write conventional unit tests for Jupyter Notebooks
  • Execute all or some specific cells before unit test
  • Share kernel context across multiple tests (using pytest fixtures)
  • Support for patching objects
  • Inject code into Jupyter notebooks
  • Works with any unit testing library - unittest, pytest or nose

Outline of the talk

  • Brief context about the landscape of Jupyter notebooks - 2 mins
  • Context/Rationale behind creating testbook - 5 mins
    • Reproducibility of explored code paths
    • Reliability of re-execution with different inputs
    • Notebooks moving from experimentation to production environments
  • Intro to testbook and its features - 15-20 mins
    • Brief demo of testbook (through images / gifs)
    • How testbook works
    • Walkthrough of key features of testbook
      • Showing simple test functions
      • Execute specific cells before test
      • Share kernel context
      • Support for patching objects
  • When to use or not use testbook (Who is testbook for) - 2 mins
  • Future/Roadmap of testbook - what we have in store for future releases - 1 min
    • Use cases for education and teaching
    • Better support of non-Python kernels

Who the talk is for

This talk is for anyone has ever worked with Jupyter Notebooks. Common users of notebooks are Data Scientists, Data Engineers, System Automation Engineers, and Teachers.

Author and maintainer at testbook. Incoming Software Engineer at Goldman Sachs, Bengaluru, India. Final year electronics engineering undergraduate at MIT Manipal, Karnataka, India.

📧 hi@rohitsanjay.com

🌐 rohitsanjay.com

Matthew is a co-founder and CTO of a startup called Noteable. Before Noteable Matthew worked with notebook and data integrations at Netflix. He's a core maintainer of many Jupyter and nteract projects such as papermill, and more recently testbook. His passions are reading, gaming, electronics, and more recently trying to teach his daughter her first words.