Advanced Python Development

This book builds on basic Python tutorials to explain various Python language features that aren’t routinely covered: from reusable console scripts that play double duty as micro-services by leveraging entry points, to using asyncio efficiently to collate data from a large number of sources. Along the way, it covers type-hint based linting, low-overhead testing and other automated quality checking to demonstrate a robust real-world development process.

Some powerful aspects of Python are often documented with contrived examples that explain the feature as a standalone example only. By following the design and build of a real-world application example from prototype to production quality you'll see not only how the various pieces of functionality work but how they integrate as part of the larger system design process. In addition, you'll benefit from the kind of useful asides and library recommendations that are a staple of conference Q&A sessions at Python conferences as well as discussions of modern Python best practice and techniques to better produce clear code that is easily maintainable.

Advanced Python Development is intended for developers who can already write simple programs in Python and want to understand when it’s appropriate to use new and advanced language features and to do so in a confident manner. It is especially of use to developers looking to progress to a more senior level and to very experienced developers who have thus far used older versions of Python.

WHAT YOU'LL LEARN

* Understand asynchronous programming


* Examine developing plugin architectures


* Work with type annotations


* Review testing techniques


* Explore packaging and dependency management



WHO THIS BOOK IS FOR

Developers at the mid to senior level who already have Python experience.

Matthew Wilkes is a European software developer who has worked with Python on web projects for the last fifteen years. As well as developing software, he has long experience in mentoring Python developers in a commercial setting.

He is also very involved in open source software, with commits to many popular

frameworks. His contributions in that space are focused on the details of database and security interactions of web frameworks.

Chapter 1: Prototyping and Environments

Chapter Goal: Create a prototype script to read a single sensor value

No of pages 25

Sub -Topics

1 Introduce the example of collating data from a large number of other

machines

1.1 Possible usecases of this pattern include log aggregation, server

monitoring, IoT, monitoring of customer servers, etc

1.2 We’ll use raspberry pis with a mix of server monitoring and

very basic sensors like temperature sensors. There’ll be no IoT

specific setup or detail, it’s just that this is easier for people to

follow along with without inventing another system being

monitored.

1.3 There’ll be plenty of extra context here for how to apply the

ongoing example if you do already have a system that needs

monitoring.

2 Prototyping using jupyter and nbconvert

3 Use pipenv to set up dependency environment

3.1 Note that by introducing pipenv before setuptools we’re

preempting the confusion about the right way to do dependency

and environment management.

Chapter 2: Testing, Checking and Linting

Chapter Goal: Progress the prototype to a series of reliable functions that

can be tested

No of pages: 30

Sub - Topics

1 Testing with PyTest (especially fixtures and MUT style)

2 Type hinting and checking with mypy

3 Linting with flake8 and autoformatting with black

4 pre-commit and commit hooks

5 GitHub CI integration for easier contributions

Chapter 3: Packaging Scripts

Chapter Goal: Create an installable package that gives a single script to

read the sensor value

No of pages : 30

Sub - Topics:

1 setup.py and setuptools when it comes to packaging (not pip /

setup.py for environment management, that’s in chapter 1)

2 Namespace packages

3 Console entrypoint

4 argparse

Helpful aside: Package name conflicts, installing from GitHub releases,

release hashing, wheels

Chapter 4: From Script to Library

Chapter Goal: Extend the package to allow reading of multiple sensors

through the command line

No of pages: 20

Sub - Topics:

1. Abstract Base Classes

2. Second sensor value

3. argparse subcommands

Chapter 5: Alternative Interfaces

Chapter Goal: Make the script functionality available as a HTTP

microservice

No of pages : 40

Sub - Topics:

1 Simple API servers using flask

2 Plugin architecture using entrypoints

3 Dynamic dispatch

4 Serialisation considerations with custom classes (like units from pints

package)

Chapter 6: Speeding Things Up

Chapter Goal: Discuss optimisation strategies, what the tradeoffs between

async and different types of caching are. We’ll use caching

here, but async later

No of pages : 25

Sub - Topics:

1 asyncio vs lru_cache vs redis vs sqlite etc

2 Use of timeit

3 File operations using context managers

Chapter 7: Aggregation Process

Chapter Goal: Create a new package, read configuration files, do a basic

HTTP loop

No of pages : 25

Sub - Topics:

1 cookiecutter

2 Config files (configparser vs json vs yaml)

3 Requests library

4 More depth in pytest usage

Chapter 8: Asynchronous Programming

Chapter Goal: Understand the event loop, especially async for loops,

demonstrate how it’s a good fit for the aggregation process

No of pages : 40

Sub - Topics:

1 Defining asynchronous functions

2 Using the event loop

3 Syntactic sugar for loops and iterators

4 Async tasks vs await

5 async executors

Chapter 9: Asynchronous Databases

Chapter Goal: Understand async executors, using sqlalchemy and JSONB

No of pages : 30

Sub - Topics:

1 sqlalchemy (and why pandas isn’t a good fit here)

2 JSONB format and schemaless

3 aiofile, asyncpg and usability/speed tradeoffs

Chapter 10: Viewing the Data

Chapter Goal: Creating Jupyter notebooks and using matplotlib

No of pages : 35

Sub - Topics:

1 Calling async functions from Jupyter Notebooks

2 Binding function calls to ipywidgets for interactive reports

3 Examples of matplotlib

4 GeoJSON

Chapter 11: Fault Tolerance

Chapter Goal: Extending ABC interfaces and efficient use of iterables for

large HTTP responses

No of pages : 20

Sub - Topics:

1 Using __subclasshook__ effectively

2 Chunked responses vs framing

3 JSON deserialisation of partial data and efficient data transfer

Chapter 12: Callbacks and Data Analysis

Chapter Goal: Using generators, iterators and coroutines for data

analysis, async timeouts

No of pages : 30

Sub - Topics:

1 Iterator based filtering

2 Coroutine based plugins, for example a coroutine that pulls historical

data and compares it to the current value to decide if an alarm should

be raised

3 waitfor and executor timeout considerations