Python-Programming-Refresher.ipynb

Kernel: Python 3 (system-wide)

Python Programming Refresher using Jupyter Notebooks

Credits

Orignal by: J.R. Johansson. An IPython notebook lecture is available at http://github.com/jrjohansson/scientific-python-lectures. Other notebooks in this lecture series are indexed at http://jrjohansson.github.io. Robert Johansson is the author of Numerical Python.

Modified and updated: C. MacNish, 2020.

---

This refresher is a dynamic document. You should run the code cells as you go through it. You can add and execute your own code to check your understanding.

References

• The Python Language Reference, https://docs.python.org/3/reference/index.html

• The Python Standard Library, https://docs.python.org/3/library/

Modules

Most of the functionality in Python is provided by modules. The Python Standard Library is a large collection of modules that provides cross-platform implementations of common facilities such as access to the operating system, file I/O, string management, network communication, and much more. To use a module in a Python program it first has to be imported. A module can be imported using the import statement. For example, to import the module math, which contains many standard mathematical functions, we can do:

import math

This includes the whole module and makes it available for use later in the program. For example, we can do:

#import math  # you don't need to import again, this is just for demonstration

x = math.cos(2 * math.pi)
print(x)

1.0

Alternatively, we can chose to import all symbols (functions and variables) in a module to the current namespace (so that we don't need to use the prefix "math." every time we use something from the math module:

from math import *

x = cos(2 * pi)
print(x)

This pattern can be very convenient, but in large programs that include many modules it is often a good idea to keep the symbols from each module in their own namespaces, by using the import math pattern. This would elminate potentially confusing problems with name space collisions.

As a third alternative, we can chose to import only a few selected symbols from a module by explicitly listing which ones we want to import instead of using the wildcard character *:

from math import cos, pi

x = cos(2 * pi)
print(x)

Looking at what a module contains, and its documentation

Once a module is imported, we can list the symbols it provides using the dir function:

import math

print(dir(math))

['__doc__', '__loader__', '__name__', '__package__', '__spec__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 'isclose', 'isfinite', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf', 'nan', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'tau', 'trunc']

And using the function help we can get a description of each function (almost .. not all functions have docstrings, as they are technically called, but the vast majority of functions are documented this way).

help(math.log)

Help on built-in function log in module math:

log(...)
    log(x[, base])
    
    Return the logarithm of x to the given base.
    If the base not specified, returns the natural logarithm (base e) of x.

log(10)

log(10, 2)

• Calculate log₁₀10

We can also use the help function directly on modules.

• Try help(math)

A complete lists of standard modules for Python 3 is available at http://docs.python.org/3/library/.

Variables and types

Symbol names

Variable names in Python can contain alphanumerical characters a-z, A-Z, 0-9 and some special characters such as _. Normal variable names must start with a letter.

By convention, variable names start with a lower-case letter, and Class names start with a capital letter. [We will also use all-caps for constants.]

In addition, there are a number of Python keywords that cannot be used as variable names. These keywords are:

and, as, assert, break, class, continue, def, del, elif, else, except, 
exec, finally, for, from, global, if, import, in, is, lambda, not, or,
pass, print, raise, return, try, while, with, yield

Note: Be aware of the keyword lambda, which could easily be a natural variable name in a scientific program. But being a keyword, it cannot be used as a variable name.

Assignment

The assignment operator in Python is =. Python is a dynamically typed language, so we do not need to specify the type of a variable when we create one.

Assigning a value to a new variable creates the variable:

# variable assignments
x = 1.0
my_variable = 12.2

Although not explicitly specified, a variable does have a type associated with it. The type is derived from the value that was assigned to it.

type(x)

If we assign a new value to a variable, its type can change.

x = 1

type(x)

If we try to use a variable that has not yet been defined we get an NameError:

print(y)

Fundamental types

# integers
x = 1
type(x)

# float
x = 1.0
type(x)

# boolean
b1 = True
b2 = False

type(b1)

# complex numbers: note the use of `j` to specify the imaginary part 
# [don't worry if you're not familiar with complex numbers]
x = 1.0 - 1.0j
type(x)

print(x)

print(x.real, x.imag)

Type utility functions

The module types contains a number of type name definitions that can be used to test if variables are of certain types:

import types

# print all types defined in the `types` module
print(dir(types))

x = 1.0

# check if the variable x is a float
type(x) is float

# check if the variable x is an int
type(x) is int

We can also use the isinstance method for testing types of variables:

isinstance(x, float)

Type casting

x = 1.5

print(x, type(x))

x = int(x)

print(x, type(x))

z = complex(x)

print(z, type(z))

x = float(z)

Complex variables cannot be cast to floats or integers. We need to use z.real or z.imag to extract the part of the complex number we want:

y = bool(z.real)
print(z.real, " -> ", y, type(y))

y = bool(z.imag)
print(z.imag, " -> ", y, type(y))

1.0  ->  True <class 'bool'>
0.0  ->  False <class 'bool'>

Operators and comparisons

Most operators and comparisons in Python work as one would expect:

• Arithmetic operators +, -, *, /, // (integer division), '**' power

(1 + 2, 1 - 2, 1 * 2, 1 / 2)

(1.0 + 2.0, 1.0 - 2.0, 1.0 * 2.0, 1.0 / 2.0)

# Integer division of float numbers
3.0 // 2.0

# Note! The power operators in python isn't ^, but **
2 ** 2

Note: The / operator always performs a floating point division in Python 3.x. This is not true in Python 2.x, where the result of / is always an integer if the operands are integers. to be more specific, 1/2 = 0.5 (float) in Python 3.x, and 1/2 = 0 (int) in Python 2.x (but 1.0/2 = 0.5 in Python 2.x).

• The boolean operators are spelled out as the words and, not, or.

True and False

not False

True or False

• Comparison operators >, <, >= (greater or equal), <= (less or equal), == equality, is identical.

(2 > 1, 2 < 1)

(2 > 2, 2 < 2)

(2 >= 2, 2 <= 2)

# equality
[1,2] == [1,2]

# identity
[1,2] is [1,2]

# objects identical?
l1 = [1,2]
l2 = l1

l1 is l2

True

• Can you explain the last two results?

Compound types: Strings, List and dictionaries

Strings

Strings are the variable type that is used for storing text.

s = "Hello world"
type(s)

# length of the string: the number of characters
len(s)

• Why does the above work?

# replace a substring in a string with something else
s2 = s.replace("world", "test")
print(s2)

• What is the difference between the type of function in the previous cell (len) and the type of function in the above cell (replace)?

(Hint: Find their documentation.)

We can index a character in a string using []:

s[0]

Heads up MATLAB users: Indexing starts at 0!

We can extract a part of a string using the syntax [start:stop], which extracts characters between index start and stop -1 (the character at index stop is not included):

s[0:5]

s[4:5]

If we omit either (or both) of start or stop from [start:stop], the default is the beginning and the end of the string, respectively:

s[:5]

s[6:]

s[:]

We can also define the step size using the syntax [start:end:step] (the default value for step is 1, as we saw above):

s[::1]

s[::2]

This technique is called slicing. Read more about the syntax here: https://docs.python.org/3/library/functions.html#slice

Python has a very rich set of functions for text processing. See for example https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str for more information.

Remember, as the documentation makes clear, str is actually a sequence type.

String formatting examples

print("str1", "str2", "str3")  # The print statement concatenates strings with a space

str1 str2 str3

print("str1", 1.0, False, -1j)  # The print statements converts all arguments (where it can) to strings

print("str1" + "str2" + "str3") # strings added with + are concatenated without space

# formatting a string [more on this later]
s3 = 'value1 = {0}, value2 = {1}'.format(3.1415, 1.5)

print(s3)

Lists

Lists are very similar to strings, except that each element can be of any type. [OK, this is kind of backwards - strings are a sequence type like lists, but let's continue...]

The syntax for creating lists in Python is [...]:

l = [1,2,3,4]  # actually it's pretty bad practice to call your list 'l', it looks a lot like '1'

print(type(l))
print(l)

We can use the same slicing techniques to manipulate lists as we could use on strings:

print(l)

print(l[1:3])

print(l[::2])

Heads up MATLAB users: Indexing starts at 0!

l[0]

Elements in a list do not all have to be of the same type:

l = [1, 'a', 1.0, 1-1j]

print(l)

[1, 'a', 1.0, (1-1j)]

Python lists can be inhomogeneous and arbitrarily nested:

nested_list = [1, [2, [3, [4, [5]]]]]

nested_list

[1, [2, [3, [4, [5]]]]]

Lists play a very important role in Python. For example they are used in loops and other flow control structures (discussed below). There are a number of convenient functions for generating lists of various types, for example the range function:

start = 10
stop = 30
step = 2

range(start, stop, step)

# in python 3 range generates an iterator, which can be converted to a list using 'list(...)'.
# It has no effect in python 2
list(range(start, stop, step))

list(range(-10, 10))

s

# convert a string to a list by type casting:
s2 = list(s)

s2

# sorting lists
s3 = sorted(s2)
print(s3)

Adding, inserting, modifying, and removing elements from lists

# create a new empty list
l = []

# add an elements using `append`
l.append("A")
l.append("d")
l.append("d")

print(l)

We can modify lists by assigning new values to elements in the list. In technical jargon, lists are mutable.

l[1] = "p"
l[2] = "p"

print(l)

l[1:3] = ["d", "d"]

print(l)

Insert an element at an specific index using insert

l.insert(0, "i")
l.insert(1, "n")
l.insert(2, "s")
l.insert(3, "e")
l.insert(4, "r")
l.insert(5, "t")

print(l)

Remove first element with specific value using 'remove'

l.remove("A")

print(l)

Remove an element at a specific location using del:

del l[7]
del l[6]

print(l)

See help(list) for more details, or read the online documentation

Tuples

Tuples are like lists, except that they cannot be modified once created, that is they are immutable.

In Python, tuples are created using the syntax (..., ..., ...):

point = (10, 20)

print(point, type(point))

point = 10, 20  # shorthand for above, but discouraged

print(point, type(point))

We can unpack a tuple by assigning it to a comma-separated list of variables:

(x, y) = point
print("x =", x)
print("y =", y)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-30-275276ffd603> in <module>
----> 1 (x, y) = point
      2 print("x =", x)
      3 print("y =", y)
NameError: name 'point' is not defined

If we try to assign a new value to an element in a tuple we get an error:

point[0] = 20

t = (10, 20)

print(t, type(t))

print(t[0])

(10, 20) <class 'tuple'>
10

Dictionaries

Dictionaries are also like lists, except that each element is a key-value pair. The syntax for dictionaries is {key1 : value1, ...}:

params = {"parameter1" : 1.0,
          "parameter2" : 2.0,
          "parameter3" : 3.0,}

print(type(params))
print(params)

print("parameter1 = " + str(params["parameter1"]))
print("parameter2 = " + str(params["parameter2"]))
print("parameter3 = " + str(params["parameter3"]))

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-34-e703c419d4c8> in <module>
----> 1 print("parameter1 = " + str(params["parameter1"]))
      2 print("parameter2 = " + str(params["parameter2"]))
      3 print("parameter3 = " + str(params["parameter3"]))
NameError: name 'params' is not defined

• Why is str needed here?

dic = {"p1" : 1.0,
       "p2" : 2.0,
       "p3" : 3.0}

print(type(dic))
print(dic)

print("The first elements is " + str(dic["p1"]))

dic["p1"] = True

print("Now the first elements is " + str(dic["p1"]))


# Add a new entry
dic["p4"] = "new"

print(dic["p4"])

<class 'dict'>
{'p1': 1.0, 'p2': 2.0, 'p3': 3.0}
The first elements is 1.0
Now the first elements is True
new

params["parameter1"] = "A"
params["parameter2"] = "B"

# add a new entry
params["parameter4"] = "D"

print("parameter1 = " + str(params["parameter1"]))
print("parameter2 = " + str(params["parameter2"]))
print("parameter3 = " + str(params["parameter3"]))
print("parameter4 = " + str(params["parameter4"]))

Control Flow

Conditional statements: if, elif, else

The Python syntax for conditional execution of code uses the keywords if, elif (else if), else:

statement1 = False
statement2 = False

if statement1:
    print("statement1 is True")

elif statement2:
    print("statement2 is True")

else:
    print("statement1 and statement2 are False")

statement1 = False
statement2 = False

if statement1:
    print("s1 is true")
elif statement2:
    print("s2 is true")
else:
    print("neither of them are true")

neither of them are true

For the first time, here we encounted a peculiar and unusual aspect of the Python programming language: Program blocks are defined by their indentation level.

Compare to the equivalent C code:

if (statement1)
{
    printf("statement1 is True\n");
}
else if (statement2)
{
    printf("statement2 is True\n");
}
else
{
    printf("statement1 and statement2 are False\n");
}

In C blocks are defined by the enclosing curly brakets { and }. And the level of indentation (white space before the code statements) does not matter (completely optional).

But in Python, the extent of a code block is defined by the indentation level (usually a tab or say four white spaces). This means that we have to be careful to indent our code correctly, or else we will get syntax errors.

Examples:

statement1 = True
statement2 = True

if statement1:
    if statement2:
        print("both statement1 and statement2 are True")

# Bad indentation!
if statement1:
    if statement2:
    print("both statement1 and statement2 are True")  # this line is not properly indented

statement1 = False

if statement1:
    print("printed if statement1 is True")

    print("still inside the if block")

if statement1:
    print("printed if statement1 is True")

print("now outside the if block")

Loops

In Python, loops can be programmed in a number of different ways. The most common is the for loop, which is used together with iterable objects, such as lists. The basic syntax is:

for loops

for x in [1,2,3]:
    print(x)

The for loop iterates over the elements of the supplied list, and executes the containing block once for each element. Any kind of list can be used in the for loop. For example:

for x in range(4): # by default range start at 0
    print(x)

Note: range(4) does not include 4 !

for x in range(-3,3):
    print(x)

-3
-2
-1
0
1
2

for word in ["scientific", "computing", "with", "python"]:
    print(word)

To iterate over key-value pairs of a dictionary:

print(params.items(),"\n")

for (key, value) in params.items():
    print(key + " = " + str(value))

Sometimes it is useful to have access to the indices of the values when iterating over a list. We can use the enumerate function for this:

for (idx, x) in enumerate(range(-3,3)):
    print(idx, x)

0 -3
1 -2
2 -1
3 0
4 1
5 2

List comprehensions: Creating lists using for loops:

A convenient and compact way to initialize lists:

l1 = [x**2 for x in range(0,5)]

print(l1)

l = [x**2 for x in range(0, 5)]

print(l)

[0, 1, 4, 9, 16]

while loops

i = 0

while i < 5:
    print(i)

    i = i + 1

print("done")

Note that the print("done") statement is not part of the while loop body because of the difference in indentation.

Functions

A function in Python is defined using the keyword def, followed by a function name, a signature within parentheses (), and a colon :. The following code, with one additional level of indentation, is the function body.

def func0():
    print("test")

func0()

Optionally, but highly recommended, we can define a so called "docstring", which is a description of the function's purpose and behaviour. The docstring should follow directly after the function definition, before the code in the function body.

[Delivery-ready code should always include a docstring for publicly accessible functions.]

def func1(s):
    """
    Print a string 's' and tell how many characters it has
    """

    print(s + " has " + str(len(s)) + " characters")

help(func1)

func1("test")

Functions that return a value use the return keyword:

def square(x):
    """
    Return the square of x.
    """
    return x ** 2

square(4)

We can return multiple values from a function using tuples (see above):

def powers(x):
    """
    Return a few powers of x.
    """
    return (x ** 2, x ** 3, x ** 4)

powers(3)

x2, x3, x4 = powers(3)

print(x3)

Default argument and keyword arguments

In a definition of a function, we can give default values to the arguments the function takes:

def myfunc(x, p=2, debug=False):
    if debug:
        print("evaluating myfunc for x = " + str(x) + " using exponent p = " + str(p))
    return x**p

If we don't provide a value of the debug argument when calling the the function myfunc it defaults to the value provided in the function definition:

myfunc(5)

myfunc(5, debug=True)

If we explicitly list the name of the arguments in the function calls, they do not need to come in the same order as in the function definition. This is called keyword arguments, and is often very useful in functions that takes a lot of optional arguments.

# In a definition of a function, we can give default values to the arguemnts
def myfunc(x, p = 2, debug = False):
    if debug:
        print("bugs there")
    return x**p

myfunc(5)
myfunc(5, 3)
myfunc(5, debug = True)
myfunc(p = 3, x = 4, debug = True)

bugs there
bugs there

64

myfunc(p=3, debug=True, x=7)

Classes

Classes are the key features of object-oriented programming. A class is a structure for representing an object and the operations that can be performed on the object.

In Python a class can contain attributes (variables) and methods (functions).

A class is defined almost like a function, but using the class keyword, and the class definition usually contains a number of class method definitions (a function in a class).

• Each class method should have an argument self as its first argument. This object is a self-reference.

• Some class method names have special meaning, for example:

  • __init__: The name of the method that is invoked when the object is first created.

  • __str__ : A method that is invoked when a simple string representation of the class is needed, as for example when printed.

  • There are many more, see http://docs.python.org/2/reference/datamodel.html#special-method-names

class Point:
    """
    Simple class for representing a point in a Cartesian coordinate system.
    """

    def __init__(self, x, y):
        """
        Create a new Point at x, y.
        """
        self.x = x
        self.y = y

    def translate(self, dx, dy):
        """
        Translate the point by dx and dy in the x and y direction.
        """
        self.x += dx
        self.y += dy

    def __str__(self):
        return("Point at [%f, %f]" % (self.x, self.y))

To create a new instance of a class:

p1 = Point(0, 0) # this will invoke the __init__ method in the Point class

print(p1)         # this will invoke the __str__ method

Point at [0.000000, 0.000000]

To invoke a class method in the class instance p:

p2 = Point(1, 1)

p1.translate(0.25, 1.5)

print(p1)
print(p2)

Point at [0.250000, 1.500000]
Point at [1.000000, 1.000000]

Note that calling class methods can modifiy the state of that particular class instance, but does not effect other class instances or any global variables.

That is one of the nice things about object-oriented design: code such as functions and related variables are grouped in separate and independent entities.

Modules

One of the most important concepts in good programming is to reuse code and avoid repetitions.

The idea is to write functions and classes with a well-defined purpose and scope, and reuse these instead of repeating similar code in different part of a program (modular programming). The result is usually that readability and maintainability of a program is greatly improved. What this means in practice is that our programs have fewer bugs, are easier to extend and debug/troubleshoot.

Python supports modular programming at different levels. Functions and classes are examples of tools for low-level modular programming. Python modules are a higher-level modular programming construct, where we can collect related variables, functions and classes in a module. A python module is defined in a python file (with file-ending .py), and it can be made accessible to other Python modules and programs using the import statement.

Consider the following example: the file mymodule.py contains simple example implementations of a variable, function and a class:

%%file mymodule.py
"""
Example of a python module. Contains a variable called my_variable,
a function called my_function, and a class called MyClass.
"""

my_variable = 0

def my_function():
    """
    Example function
    """
    return my_variable

class MyClass:
    """
    Example class.
    """

    def __init__(self):
        self.variable = my_variable

    def set_variable(self, new_value):
        """
        Set self.variable to a new value
        """
        self.variable = new_value

    def get_variable(self):
        return self.variable

We can import the module mymodule into our Python program using import:

import mymodule

Use help(module) to get a summary of what the module provides:

help(mymodule)

mymodule.my_variable

mymodule.my_function()

my_class = mymodule.MyClass()
my_class.set_variable(10)
my_class.get_variable()

Exceptions

In Python errors are managed with a special language construct called "Exceptions". When errors occur exceptions can be raised, which interrupts the normal program flow and fallback to somewhere else in the code where the closest try-except statement is defined. To generate an exception we can use the raise statement, which takes an argument that must be an instance of the class BaseException or a class derived from it.

raise Exception("description of the error")

A typical use of exceptions is to abort functions when some error condition occurs, for example:

def my_function(arguments):

    if not verify(arguments):
        raise Exception("Invalid arguments")

      rest of the code goes here

# The first way to generate an exception
def func(a):
    
    if not verify(a):
        raise Exception("Invalid arguments")
    
    ...

# The second way to generate an exception
def func(b):
    
    try:
        normal code that might has error 
    except:
        if there are some errors for the code in try,
        this code would be generated 

To gracefully catch errors that are generated by functions and class methods, or by the Python interpreter itself, use the try and except statements:

try:
    normal code goes here
except:
    code for error handling goes here
    this code is not executed unless the code
    above generated an error

For example:

try:
    print("test")
    # generate an error: the variable test is not defined
    print(test)
except:
    print("Caught an exception")

To get information about the error, we can access the Exception class instance that describes the exception by using for example:

except Exception as e:

try:
    print("test")
    # generate an error: the variable test is not defined
    print(test)
except Exception as e:
    print("Caught an exception:" + str(e))

Further reading

If you wish to refer back to a topic in this file, you can use the Table of Contents. It can be found in the 'Notebook' drop-down menu at the top right of the workspace. The Table of Contents entries are hyperlinks back to the notebook.

For further reading:

• http://www.python.org - The official web page of the Python programming language.

• https://greenteapress.com/wp/think-python-2e/ - A free book on Python programming.