Lab 1
Diversity in Study: What (Else) Are We All Studying?
Data Science Across the Disciplines
Data Science has application right across the disciplines. In this course we'll see case studies ranging from the Environment to Economics to Health.
Before we start this journey, it's interesting to see what diverse areas the group is studying. As we said in the Welcome lecture, there are many students in the unit with many different study backgrounds and focusses.
To get a feel for this, we'll look at the range of enrolment patterns for this year.
Note that some randomisation has been applied to the enrolment patterns to ensure privacy and anonymity. However the disciplines are preserved.
Reference links
In this lab we will practise reading in data from a file, cleaning the data, and observing features of the data.
We'll be using standard python. The latest documentation will be useful, and it is suggested you bookmark it for future use.
The Python Language Reference, https://docs.python.org/3/reference/
The Python Standard Library, https://docs.python.org/3/library/
The Python Tutorial, https://docs.python.org/3/tutorial/
Data Acquisition
In future labs we'll be hunting down data as part of the work. In this case however the data will be provided in a text file.
Open the directory from the folder icon. Click on the file
unit-patterns.txtto inspect the file.
The file contains all the different study plans of enrolled students (at the time of writing) with some obfuscation for privacy and anonymity.
This is a tab-separated values (tsv) file with two fields.
The first field contains a list of units studied (separated by plus signs), the second (after the tab character) contains a number and a percentage.
Note that in this lab we will refer to an instance of a unit name combined with a teaching period as a 'unit' - so for example, CITS1401-1 and CITS1401-2 would be treated as two distinct units.
Let's start with a constant to identify the file. By convention we'll use uppercase for constants. Run the following cell to make the assignment.
As we saw in the 'Getting Started' lab, we can check to see that the assignment was made just by typing the variable name as the last thing in the cell and running the cell. Try this now.
Reading the file
Use the built-in function
open()and the text file IO methodreadline()to read in and print out the first 5 lines of the file.
Your output should start like this:
Tip: If you're a little rusty on reading files, you may find this tutorial useful: 7.2. Reading and Writing Files.
Also the Case Study video "CITS2402 FakeNews 2" provides a couple of examples of working with a tsv file.
Read the file again, but this time, as you read the lines from the file, save them in a list called
pattern-strings.
Printing the first 5 lines of your list should give:
Tip: Rather than using readline(), use an iterator on the file directly to get each line in turn.
Data Cleaning
We are only interested in the patterns to the left of the tab.
Using the string
partition()method, modify your code so thatpattern_stringscontains only the unit patterns.
The documentation for String Methods may be helpful.
Printing the first 5 lines of your list (print(pattern_strings[:5])) should now give:
How many students were enrolled at the time the list was taken?
Rather than the units in each line being part of a string, it is more useful to have them in a list. This will enable us to iterate through the list later.
Use the
splitmethod to split each line into a list of units. Store the result in a variablepattern_lists.
This should result in a list of lists. Your output should start like this:
Finally, notice that we've been left with unnecessary whitespace around each unit name.
Modify your code to strip off the extra whitespace.
Your first 5 patterns should now look like this:
Putting it all together
Congratulations, you have now 'wrangled' your data from its 'raw' format in the file, into a very useable form.
To consolidate this section, turn your code into a function
get_patterns(filename)that takes a filename, and returns a list of lists containing all the patterns in the file.
print(get_patterns(DATAFILE)[:5]) should now print the same output as above.
Checked Questions and Answers
Many of the labs this semester will have one or two checked (or 'autograded') answers. These will contribute to the (35%) practical marks for the semester.
The checked answers will generally have two parts, or two cells.
Answer cells
The first cell is where you enter your code to be checked. This may contains a "stub" - that is, a part of the code to be completed. For example, it may be a function definition.
Test cells
The second cell is where the code to check the answer will go and is not editable. It may be blank, but will usually contain some example tests that you can use to test your own code before the answers are collected. There will be additional tests applied that are not shown.
Collection and deadlines
The notebook will be collected and marked, by running the tests, after the lab deadline. (As the same notebook will be collected, you may wish to take a backup copy of the notebook before starting work. Alternatively you can use the TimeTravel feature to revert the notebook if you need to start again.)
Everyone will automatically receive a one week "extension" from the week of the lab to finish off the work and to account for any unforeseen circumstances.
The deadline for credit will therefore be 11:59pm on the Friday of the week after the lab is released.
Checked Solution [1 mark]
Copy your solution for
get_patterns(filename)to complete the following stub.
(We wouldn't normally enter the function a second time, it is only because we are introducing the graded answer for the first time.)
Before a final test of your code, it is worth restarting the kernel from the "Kernel" drop-down menu, and re-running your code. This will clean up any leftover 'junk' in the memory from running earlier versions of bits of code.
You may find, for example, that after you fixed or changed a piece of code, you forgot to change a variable name, but the code still worked because an old value for the variable was still in memory.
Restart the kernel and test your code again to make sure it is doing everything correctly.
Running the grading tests
The test cells will often include some 'non-hidden' tests that you can run to see if your code passes. There will also be some 'hidden' tests that are only run once the code is collected after the deadline.
If your code produces the correct answers it will simply run successfully.
If your code does not pass the non-hidden tests, you will normally get an error message. For example, if I only stripped the whitespace from the left side and not both sides of the unit names, I might get an error message like this:
The error messages may vary in their usefulness - they are not intended as a debugging tool, but often they will give some pointer as to what went wrong. In this case it is fairly clear that there is whitespace that shouldn't be there.
You can then correct your code and try again.
The following test is worth 1 prac mark. Give it a try. You can run the tests as may times as you wish before the deadline.
Data Inspection and Interpretation
Now let's have a closer look at the data. The file contains all the different study plans of enrolled students. Let's start by working out how many different or unique plans there are.
To make it easier to see what is going on, start by defining a better print function.
Write a function
print_patterns (patterns, numlines)that prints the firstnumlinespatterns each on a new line.
For example:
print_patterns(get_patterns(DATA), 8)
Write a function that takes a pattern list, and returns the list in which:
the units in each list (pattern) are sorted alpha-numerically
the lists (patterns) themselves are sorted alpha-numerically
The following test code:
should produce the following output:
Caution: If you are using python's built-in sorting methods, it is highly recommended that you use sorted to return a new sorted list, rather than sort to sort the list in-place. Methods that alter a data structure 'in-place' can lead to confusing behaviour when debugging.
Write a function
remove_duplicates (patterns)that returns a (sorted) list of patterns with any duplicate patterns removed.
How many unique study patterns are there in the class?
Print the percentage (to one decimal place) of the total number of study plans in the class that are different.
You should find that we are indeed a diverse group!
Checked Solution [1 mark]
Write a function
get_unique_patterns (patterns)that returns a pair(unique_patterns, num_patterns)where:unique_patternscontains a sorted list of sorted patterns, with any duplicate patterns removed (as above)num_patternscontains the number of unique patterns
Examining the range of disciplines
For this part, it is left to you to think about the best way to break the problems down. You won't want to write new code for each different discipline, so think about how you can use structured programming to efficiently answer the questions.
What proportion of study patterns contain:
a maths unit?
a stats unit?
a business unit?
a medicine unit?
a law unit?
a psychology unit?
a music unit?
a philosophy unit?
a service learning unit?
How many different disciplines are studied altogether (for the purposes of this question assume the first four characters of the unit code represent the discipline.)
Challenge
Print a list of each unit that is being studied, and how many students are studying it, in order from the unit with the most students to the unit with the least.
Hints: Use a dictionary, look at the get method, and consider the key argument of the inbuilt function sorted.
(My code is 7 lines including the print statement.)
How many units are being studied altogether?