code bits and other stuff by Miguel Lavalle

On being pythonic

Miguel Lavalle — Thu, 25 Nov 2021 23:17:30 GMT

I recently was given a Python coding challenge during a job interview. I was criticized for not writing "pythonic" enough code. The good news is that I did great with the other interviewers in the panel, so I still got the job \o/. Afterwards, though, I decided to use the criticism as a learning opportunity, revisit the challenge and write this blog post with the resulting code, while making my best effort to be "pythonic". To that end, here's a basic definition: pythonic code uses the powerful idioms of Python that distinguishes it from other languages to produce elegant and concise programs. To guide me on the road to being "pythonic", I am using the following references:

Transforming Code into Beautiful, Idiomatic Python (slides) by Raymond Hettinger, who is a Python core developer.
Since in this exercise we are going to use generators and coroutines, my other reference is Making sense of generators, coroutines, and “yield from” in Python by Reuven Lerner, who teaches Python all over the world.

In the code that follows I will cover the following topics

One of the aspects that strongly distinguishes Python from other programming languages is its approach to iterations. As Raymond Hettinger argues in his presentation, Python's for statement should have been called for each to better reflect its semantics. I will talk about iterables, iterators, the iterator protocol, generators and co-routines.
Python's regular expresions module.
Improving clarity with named tuples.
Factoring out administrative logic with decorators.

The coding problem was stated as follows:

In [46]:

# Model a computing device that can load third party programs.

# PROGRAMS consist of ACTIONS that share an execution ENVIRONMENT,
# particularly computer STATE. ACTIONS belong to a limited set supported
# by this device. ACTIONS can communicate with each other through
# REGISTERS.

# PROGRAM execution results in computer STATE change. The STATE change
# can then be inspected.

# Device should allow to LOAD multiple PROGRAMS and execute them
# SEQUENTIALLY.

# Typical API of the computing device would include:
# - load(_program)
# - execute
# - (get_)state

# Implement the device, define two programs (see below), load them
# and execute, then show results.

# program1: receive int through INPUT; add +10 to the INPUT; store result.
# program2: load result of previous program execution; depending on if
#           it's > 50, set a register in machine state to 100; otherwise to 0.

# usage example
# c = Computer(...)
# p1 = Program(...)
# p2 = Program(...)
# c.state()  # -> {...}
# c.load(p1, input=...)
# c.load(p2, input=...)
# c.execute()
# c.state()  # -> {...0 or 100...}

Recognizing instructions with Python regular expressions¶

To make parsing programs easy, the computing device (henceforth the computer) will have one operand instructions and an accumulator register. The add instruction, for example, adds its operand to the accumulator register. This means that we are also going to need lda and sto instructions, that load a value to the accumulator and store the value of the accumulator in another register, respectively. Those registers are part of the computer's state and are described further below. To be able to write program1 and program2 in the coding challenge, the following is the minimum action set necessary:

In [47]:

ACTION_SET = set(['lda', 'sto', 'bgt', 'cmp', 'add'])
ACTIONS_NO_LITERAL = set(['sto', 'bgt'])
BRANCHES = set(['bgt'])

Some instructions, like lda or add can take two types of operands, registers and literals:

lda r0 loads the content of register r0 to the acumulator.
lda #10 loads the value 10 to the acumulator.

In the case of the sto instruction, though, it doesn't make sense to have literal operands. This instruction always stores the value in the acumulator in one of the registers. The set ACTIONS_NO_LITERAL will help in processing this type of instructions.

Similarly, branches only accept destination operands, which is an integer i from 0 to n - 1, where n is the total number of instructions in the program. Branches, like bgt, direct the computer to continue fetching instructions from the location specified by the destination operand. In other words, the computer is expected to fetch next the ith instruction from the beginning of the program. The set BRANCHES will help in processing this type of instructions.

To parse a programs, we are going to use Python's re (regular expressions) module:

In [48]:

import re

RE_LINE = re.compile(
    r'^(?P<action>[a-z]{3})\s+(?P<arg>r[io0-9]$|\#-?[0-9]+$|[0-9]+$)')

This regular expression parses one instruction at a time. It can be divided in three parts:

The named capture group ^(?P<action>[a-z]{3}) recognizes sequences of 3 alphabetic characters that correspond to members of ACTION_SET. Note that it is anchored to the beginning of the string with ^.
One or more whitespaces \s+ that separate the action from the arg.
The named capture group (?P<arg>r[io0-9]$|\#-?[0-9]+$|[0-9]+$). This group can be one of three alternatives, separated by | in the regular expression (note that each alternative is anchored to the end of the string with $):

r[io0-9]$, one of the computer's registers ri, ro, r0,... r9 or...
#-?[0-9]+$, a literal beginning with # as explained above or...
[0-9]+$, a branch destination as explained above.

These are a few examples of the instructions that can be parsed:

In [49]:

# Instruction with register operand
match = RE_LINE.search('lda r0')
print(match.group('action'))
print(match.group('arg'), '\n')

# Instruction with literal operand
match = RE_LINE.search('add      #16')
print(match.group('action'))
print(match.group('arg'), '\n')

# Instruction with branch destination operand
match = RE_LINE.search('bgt 23')
print(match.group('action'))
print(match.group('arg'))

lda
r0 

add
#16 

bgt
23

If the provided string doesn't conform with the regular expression, no match object will be returned by the search method:

In [50]:

# Invalid action
match = RE_LINE.search('ldab r0')
print(match, '\n')

# Invalid operand
match = RE_LINE.search('lda ##10')
print(match, '\n')

# No operand
match = RE_LINE.search('lda')
print(match)

None 

None 

None

For a deeper look into Python's regular expressions, here's part 1 and part 2 of an excellent tutorial.

Using coroutines to implement programs¶

Python lists have two characteristics that make them very suitable to represent the programs as defined in this coding challenge:

They are an ordered collection of objects. A program is an ordered collection of instructions, with each instruction being a string that can be recognized using the regular expresion RE_LINE defined above.
Their individual elements can be directly accesed using and integer index. This index will serve as the program counter, i.e. the pointer to the next instruction within a program to be executed.

We are going to tweak the behavior of lists a little bit, though, to control better the execution of programs. So instead of using lists directly, we are going to use collections.UserList, a useful base class for user defined list-like classes which can inherit from them and override existing methods or add new ones. To be more precise, there will be a Program class that will inherit from collections.UserList and override the __iter__() method. As can be seen below, the original __iter__() method initializes an index i with 0 and proceeds to iterate sequentially over the list, returning the next element in each iteration:

In [51]:

import collections
import inspect

print(inspect.getsource(collections.UserList.__iter__))

    def __iter__(self):
        i = 0
        try:
            while True:
                v = self[i]
                yield v
                i += 1
        except IndexError:
            return

The __iter__() method in our Program class will be a coroutine that will enable the implementation of branch instructions by changing the value of the index variable to any value between 0 and the length - 1 of the program. To see how, let's take a little detour to review in more detail some Python concepts and constructs.

The __iter__() method defined by collections.UserList is a generator function. As explained in Making sense of generators, coroutines, and “yield from” in Python:

A generator function, when executed, returns a generator object.
A generator object implements the iterator protocol, meaning that it knows what to do in a for loop.
Each time a generator object reaches a yield statement, it returns the yielded value to the for loop that is invoking it, and goes to sleep.
With each successive iteration, the generator object starts running from where it paused (i.e., just after the most recent yield statement)
When the generator object reaches the end of the function, or encounters a return statement, it raises a StopIteration exception, which is how Python iterators indicate to for loops that they’ve reached the end of the line.

Let's define a collections.UserList and iterate over it with a for loop:

In [52]:

a_list = collections.UserList([0, 1, 2])

for item in a_list:
    print(item)

0
1
2

We can also interact with the generator function directly, similar to what the for statement does behind the scenes. An iterable object, such as a collection.UserList, produces a fresh new iterator or generator object each time you pass it to the iter() built-in function

In [53]:

a_list = collections.UserList([0, 1, 2])

g = iter(a_list)
g

Out[53]:

<generator object Sequence.__iter__ at 0x7f121e2693c0>

Notice that when iter() was called, it returned a generator object. This of course happened because iter() in turn called the __iter__() in collections.UserList. We now execute the generator object with the next() built-in function:

In [54]:

print(next(g))
print(next(g))
print(next(g))

try:
    next(g)
except StopIteration:
    print('End of a_list')

0
1
2
End of a_list

Generator functions use the statement yield to produce a stream of objects so their callers can iterate over them (in a for loop for example). yield, though, can also be an expression and be placed on the right hand side of an assigment. This allows functions to consume values sent to them with their send() method by their callers. Such functions are called coroutines and open the possibility of concurrent programming in a single thread. Let's look at a simple example, borrowed from A Curious Course on Coroutines and Concurrency:

In [55]:

def grep(pattern):
    print ("Looking for %s" % pattern)
    while True:
        line = (yield)
        if pattern in line:
            print(line)

Execution of a coroutine is similar to a generator. When a coroutine is called, it returns a generator:

In [56]:

c = grep("python")
c

Out[56]:

<generator object grep at 0x7f121e269580>

Coroutines only run in response to the next() statement (like generators) and their send() method. All coroutines must be "primed" first by calling next() on them. This advances execution to the first yield expression. At this, point a coroutine is ready to receive a value:

In [57]:

# Prime the coroutine so execution advances to the yield expression
next(c)

Looking for python

In [58]:

# Send lines to the coroutine, so it can search for the pattern "python"
c.send("Yeah, but no, but yeah, but no")
c.send("A series of tubes")
c.send("python generators and coroutines rock!")

python generators and coroutines rock!

As indicated above, our Program class inherits from collections.UserList and refines it as follows:

It overrides the __iter__() method with a coroutine. This enables the Computer class that we will define below to iterate sequentially over the instructions in a Program, while making it possible to change the next instruction to be returned as a result of a branch instruction execution. An int value produced by the coroutine's yield expression is used as a pointer to the next instruction to be retrieved, thus altering the flow of execution.
It adds a _parse() method, which parses each instruction using the regular expression presented above, performs additional checks and returns a ParsedInstruction named tuple with two attributes, action and argument, for ease of execution by the Computer class.

In [59]:

s = collections.namedtuple("ParsedInstruction",
                           "action, argument")


class Program(collections.UserList):

    def __iter__(self):
        next_instruction = 0
        try:
            while True:
                instruction = self[next_instruction]
                next_instruction += 1
                branch_dest = yield self._parse(instruction)
                
                # Flow of execution is altered when the preceding yield
                # expression produces a value of type int
                if isinstance(branch_dest, int):
                    next_instruction = branch_dest
        except IndexError:
            return

    def _parse(self, instruction):
        match = RE_LINE.search(instruction.lower())
        if not match:
            raise ValueError('Invalid instruction "%s"' % instruction)
        action = match.group('action')
        arg = match.group('arg')
        is_arg_literal = arg[0] == '#'
        is_branch_destination = arg.isnumeric()
        if action not in ACTION_SET:
            raise ValueError('Action "%s" not in actions set' %
                             match.group('action'))
        if is_arg_literal:
            if action in ACTIONS_NO_LITERAL:
                raise ValueError('Action "%s" cannot have a literal argument' %
                                 match.group('action'))
            arg = int(arg[1:])
        if action in BRANCHES:
            if not is_branch_destination:
                raise ValueError(
                    'Branch "%s" at %s has non branch destination "%s"' %
                    (action, self._next, arg))
            arg = int(arg)
            if arg >= len(self) or arg < 0:
                raise ValueError(
                    'Branch "%s" to destination "%s" jumps outside program '
                    'boundaries' % (action, arg))
        return ParsedInstruction(action, arg)

Let's create a Program and its generator (coroutine) and "prime" the latter to start fetching instructions:

In [60]:

# Define a program with 3 instructions
p = Program(['add #10',     # add 10 to the accumulator
             'cmp #50',     # if the accumulator is greater than 50...
             'bgt 0'])      # ... loop back to the first instruction

# Get the Program's iterator (coroutine)
coroutine = iter(p)

# "Prime" the coroutine and fetch the first instruction
parsed_instruction = next(coroutine)
parsed_instruction

Out[60]:

ParsedInstruction(action='add', argument=10)

We now fetch the other two instructions in the program and, for the bgt instruction, alter the flow of execution by sending a 0 (the first instruction's location) back to the coroutine:

In [61]:

# Fetch the cmp instruction
parsed_instruction = coroutine.send(None)
print(parsed_instruction)

# Fetch the bgt instruction
parsed_instruction = coroutine.send(None)
print(parsed_instruction)

# Branch back to the add instruction at location 0
parsed_instruction = coroutine.send(0)
print(parsed_instruction)

ParsedInstruction(action='cmp', argument=50)
ParsedInstruction(action='bgt', argument=0)
ParsedInstruction(action='add', argument=10)

Using decorators to factor-out logic¶

By definition, a decorator is a function that takes another function as argument and extends the behavior of the latter function without explicitly modifying it. This makes decorators a great mechanism to factor-out logic that is repeated across several functions or methods. Frequently, this is "administrative" logic that is not part of the main functionality that we are implementing. In our case, we are implementing two decorators that are used by the Computer class defined below:

As mentioned above, many Computer instructions can take register or literal operands. In the Computer class we have, for each instruction, a method that implements it. The methods for instructions that can take both types of operands are decorated with get_argument. This decorator retrieves the actual value to be operated on from the register specified in the instruction, when that is the case. This way, the methods can assume they only receive literal operands.
For debugging purposes, we may choose to log to the console the name of methods that implement instructions as they are executed, along with the arguments received. To accomplish this, we use the log decorator together with a LOG flag that is configured with True or False in the Computer class.

In [62]:

import functools


def get_argument(func):
    functools.wraps(func)

    def wrapper(self, argument):
        to_pass = argument
        if not isinstance(argument, int):
            to_pass = self._state[argument]
        return func(self, to_pass)

    return wrapper


def log(func):
    functools.wraps(func)

    def wrapper(self, argument):
        if LOG:
            print(func.__name__, argument)
        return func(self, argument)

    return wrapper

Implementing the computer¶

We can now implement the Computer class. Its core is the execute() method, which iterates over the instructions of each program by interacting with the Program's __iter__() coroutine in the same manner we interacted with it manually above. For each parsed_instruction yielded by the coroutine, execute() passes parsed_instruction.action as input to Python's getattr() to select and execute the specific method that implements the instruction, which receives parsed_instruction.argument as input. Most of the methods that implement instructions return None to execute(), which in turn sends it back to the coroutine. The exceptions are the branch instructions, of which we only have bgt. These branch methods return the integer representing the location of the next instruction to be executed.

Following the iterator protocol, execute() catches the StopIteration exception, which is the mechanism that the coroutine uses to signal the end of a program.

In [63]:

LOG = True 


class Computer(object):
    def __init__(self):
        self._state = {
            "ri": None,
            "ro": None,
            "r0": None,
            "r1": None,
            "r2": None,
            "r3": None,
            "r4": None,
            "r5": None,
            "r6": None,
            "r7": None,
            "r8": None,
            "r9": None,
            "zero": None,
            "negative": None,
            "accumulator": 0
        }

        self._programs = []
        self._inputs = []

    def load(self, program, input=0):
        self._programs.append(iter(program))
        self._inputs.append(input)

    def execute(self):
        for i, program in enumerate(self._programs):
            self._state['ri'] = self._inputs[i]
            parsed_instruction = next(program)
            while True:
                branch_dest = getattr(self, parsed_instruction.action)(
                    parsed_instruction.argument)
                try:
                    parsed_instruction = program.send(branch_dest)
                except StopIteration:
                    # End of program
                    break

    @get_argument
    @log
    def lda(self, argument):
        self._state['accumulator'] = argument

    @log
    def sto(self, argument):
        self._state[argument] = self._state['accumulator']

    @log
    def bgt(self, argument):
        if not (self._state['zero'] or self._state['negative']):
            return argument

    @get_argument
    @log
    def cmp(self, argument):
        v = self._state['accumulator'] - argument
        # Set flags in the computer state that are used by the logic
        # of branch methods
        self._state['zero'] = not v
        self._state['negative'] = v < 0

    @get_argument
    @log
    def add(self, argument):
        self._state['accumulator'] += argument

    def state(self):
        return self._state

We can now define a function that creates a computer and the two programs defined by the coding challenge above:

In [64]:

def create_computer_and_programs():
    c = Computer()

    # p1: receive int through INPUT; add +10 to the INPUT; store result.
    p1 = Program(['lda ri',     # load input
                  'add #10',    # add 10 to it
                  'sto r0'])    # store result for next program

    # p2: load result of previous program execution; depending on if
    #     it's > 50, set a register (r1) in machine state to 100; otherwise to 0.
    p2 = Program(['lda #100',   # assume p1 output > 50, so...
                  'sto r1',     # ... set r1 to 100
                  'lda r0',     # load p1 output
                  'cmp #50',    # if p1 output...
                  'bgt 7',      # ... is greater than 50, r1 already set up
                  'lda #0',     # p1 output is less than 50 so ...
                  'sto r1',     # ... set r1 to 0
                  'cmp #0'])    # noop to have destination for bgt above
    return c, p1, p2

If we load p1 with an input value of 40, its output (stored in r0) will be 50. As a consequence, p2 will set r1 to 0:

In [65]:

c, p1, p2 = create_computer_and_programs()
c.load(p1, input=40)
c.load(p2)
c.execute()
print("p2's output = %s" % c.state()['r1'])

lda 40
add 10
sto r0
lda 100
sto r1
lda 50
cmp 50
bgt 7
lda 0
sto r1
cmp 0
p2's output = 0

Whereas if p1 is loaded with an input value of 41, its output will be 51 and p2 will set r1 to 100:

In [66]:

c, p1, p2 = create_computer_and_programs()
c.load(p1, input=41)
c.load(p2)
c.execute()
print("p2's output = %s" % c.state()['r1'])

lda 41
add 10
sto r0
lda 100
sto r1
lda 51
cmp 50
bgt 7
cmp 0
p2's output = 100

With this we conclude the implementation of this coding challenge.

Implementing multiple bindings for OpenStack Networking ports

Miguel Lavalle — Fri, 08 Dec 2017 21:53:16 GMT

One would be hard pressed to point out a more fundamental function of OpenStack Networking (a.k.a. Neutron) than that of providing virtual ports and the process of binding them. It is through bound ports that virtual machines (instances) and higher level services like load balancers can access the virtual networking provided by Neutron. In this blog post I want to provide an overview of that process we call port binding and then explain how and why we are implementing multiple port bindings to better support the migration of Nova instances. This work is being done for the ML2 plug-in, which is the reference Neutron core plug-in. The upshot is that all the ML2 based mechanism drivers that are not part of the Neutron code repository will also be enabled with multiple port bindings.

ML2 port binding

A port is an access point to a Neutron virtual network. Virtual machines, bare metal servers and higher level services use ports to send and receive data over a virtual network. Binding is the process whereby the ML2 core plug-in decides how a port is going to be connected physically to the network to which it belongs.

One of the key design goals for ML2 was to support heterogeneous networking technology across compute nodes. A simple example can be a deployment where some of the compute nodes use OVS while others use Linux Bridge. To support such heterogeneity, ML2 offers the concept of mechanism drivers. In our example deployment, we would configure two mechanism drivers: the OVS driver and the Linux Bridge driver. A mechanism driver is responsible of configuring the physical infrastructure (OVS or Linux Bridge in our example) to bind ports to that infrastructure so they can access their virtual network. The binding process has a set of well defined inputs:

Port attributes.
- binding:host_id. This is a string specifying the name of the host where the port should be bound.
- binding:vnic_type. A Neutron port can be requested to be bound as a virtual NIC (OVS, Linux Bridge) , direct pci-passthrough, direct macvtap or other types. A mechanism driver only binds a port if it supports its vnic_type.
- binding:profile. This is a dictionary of key / value pairs that provides information used to influence the binding process. ML2 will accept, store, and return arbitrary key / value pairs within the dictionary and their semantics are mechanism driver dependent.
Network infrastructure configuration. ML2 carries out the process of binding a port outside any DB transaction. This is to allow the mechanism drivers to configure the infrastructure by performing blocking calls.

ML2 will attempt a binding when, during the processing of a ReST API port create or update call, it finds that binding:host_id is defined (not None) and binding:vif_type has a value of VIF_TYPE_UNBOUND or VIF_TYPE_BINDING_FAILED. Under these conditions, ML2 calls the bind_port method of each mechanism driver in the order in which they are configured in the mechanism_drivers option in the ml2 section of /etc/neutron/plugins/ml2/ml2_conf.ini. This process continues until one of the drivers binds the port successfully to one of the network segments or all the drivers have been called and have failed, in which case binding:vif_type is set to VIF_TYPE_BINDING_FAILED. Failure to bind doesn't mean the ReST API call fails. A port create or update can still succeed even though it was not possible to bind it.

There is much more to be said about port binding that goes beyond the scope of this post. To learn more about the subject, you can watch the master on the subject, Robert Kukura, giving a presentation during the 2016 OpenStack Summit in Austin 1.

Better support for instance live migration with multiple port bindings

Instance live migration consists of three stages:

pre_live_migration. Executed before migration starts. The target host is determined in this stage, but the instance still resides on the source.
live_migration_operation. This is the stage where the instance is moved to the target host.
post_live_migration. The migration has concluded and the source instance doesn't exist anymore. Up until now, this is the stage where the instance ports were bound.

The problem with this flow is that the port binding happens too late:

If the binding fails, all the migration steps taken up to this point are wasted and the instance gets stuck in an error state. If the migration is going to fail due to port binding, we want it to happen as early as possible in the process, preferably before the instance is migrated.
Network downtime is lengthened by binding the ports in the post_live_migration stage, since the source instance is removed before the binding starts.
The destination instance definition is built using the results of the source instance bindings (vif_type and vif_details). This prevents the possibility of migrating the instance between hosts with different networking technology, from OVS to Linux bridge for example (or to a new and promising technology).

To address these issues, the destination instance port binding creation needs to be moved to the pre_live_migration stage. But since the source instance is still alive at this stage and using its ports bindings, we need to be able to associate more than one binding with each port. The outline of the solution is the following:

A port can have more than one binding. Each binding corresponds to a specific host.
Only one binding will be in PORT_BINDING_STATUS_ACTIVE. The others will be in PORT_BINDING_STATUS_INACTIVE.
The Neutron ReST API is extended to support CRUD operations for multiple port bindings. Also an activate binding operation is added.
When live migrating an instance, Nova will use the new ReST API extension to:
- Create during pre_live_migration new bindings in PORT_BINDING_STATUS_INACTIVE.
- Use information gathered from the inactive binding to modify the instance definition during the live_migration_operation.
- Use the activate operation to set the source instance bindings to PORT_BINDING_STATUS_INACTIVE and the destination instance bindings to PORT_BINDING_STATUS_ACTIVE once the latter instance becomes active during the live_migration_operation.
- Remove the inactive bindings on the source compute host during the post_live_migration.

The specific calls in the new ReST API extension are the following (for details on the calls requests and responses please see 2):

POST /v2.0/ports/{port_id}/bindings. The request body has to specify the host to which the binding will be associated and can specify optionally a vnic_type and a profile. The call returns vif_type and vif_details and results in a 500 code if the binding fails. In the current implementation, only instance ports (device_owner == const.DEVICE_OWNER_COMPUTE_PREFIX) can have multiple port bindings and a maximum of 2 bindings are allowed per port.
PUT /v2.0/ports/{port_id}/bindings/{host_id}. Allows the caller to update the vnic_type and profile. It returns a new vif_type and vif_details and results in 500 if the binding fails.
PUT /v2.0/ports/{port_id}/bindings/{host_id}/activate. When applied to an inactivate binding, it will activate it and inactivate the previously active one. Attempting to activate an existing active binding will return a 400. It will return a 500 if the binding fails.
GET /v2.0/ports/{port_id}/bindings. Returns the bindings associated to a port.
GET /v2.0/ports/{port_id}/bindings/{host_id}. Returns the details of a specific binding.

Conclusion

I have given an overview of the port binding process and how it is being improved to better support instance live migration. This is the latest chapter in a long history of cooperation between the Neutron and Nova teams that will continue in the future in our joint quest to better support OpenStack users. Please see the specification in 3 to learn how the Nova team is leveraging multiple port bindings.

References

1: Understanding ML2 Port Binding: https://www.youtube.com/watch?v=e38XM-QaA5Q&t=1801s
2: Provide Port Binding Information for Nova Live Migration specification: https://specs.openstack.org/openstack/neutron-specs/specs/backlog/pike/portbinding_information_for_nova.html
3: Use Neutron’s new port binding API specification: https://specs.openstack.org/openstack/nova-specs/specs/queens/approved/neutron-new-port-binding-api.html