| Download
Draft of article for the Notices of the AMS called "Collaborative Calculation in Your Web Browser"
Project: math research - wstein
Views: 453Image: ubuntu2004
\documentclass{article}12\usepackage{hyperref}34\title{Collaborative Calculation in Your Web Browser}5\author{William Stein}67\begin{document}8\maketitle910I greatly appreciate the opportunity to write about online resources for11mathematics in this theme of the Early Career Section, since such resources are12something that I have long enjoyed sharing with the community. I first started13creating websites for number theorists in the late 1990s when I was a graduate14student at UC Berkeley. I started by creating tables of modular forms, inspired15by a question from Ken Ribet, then encouraged by other people who told me that16my data inspired their research (e.g., \cite{emerton}). I later became involved17with the $L$-functions and Modular Forms Database (LMFDB) project \cite{lmfdb}, which organized dozens of mathematicians to18make amazing tables of number theoretic data.1920I also created interactive online calculators, in which you could input the21parameters of some mathematical object (e.g., an elliptic curve) into a web22page, and the web server would compute and display extensive information. Later,23when teaching undergraduate number theory courses at Harvard, I made similar24online calculators that enabled my students to run calculations using PARI and25Magma (see \cite{magma}), so that they could explore nontrivial26computations in algebraic number theory. I wanted something similar to Magma27that was free and open source, so I started the Python-based software SageMath28in 2004 (see \cite{sage}).2930SageMath was difficult to install and only had a command line interface, so I31started working to make it possible to use in a web browser. At University of32Washington (UW) in 2006, some students and I undertook the creation of a33web-based interactive notebook interface called the Sage Notebook, which was34inspired by Mathematica's desktop notebook. The Sage Notebook was the first35serious web-based computational notebook, and it was challenging for us to36implement because of the primitive Javascript and HTML technology of the day,37and the dangers arising from running arbitrary user code on my server. I hosted38the notebook publicly for anybody to use starting in 2007, and my students and39others frequently used it in courses, summer student research programs, and40SageMath development workshops. A few years later the IPython project created a41new notebook called Jupyter that looked and felt similar, but with a more42modern underlying architecture that was built to work with a wide range of43programming environments. The arrival of the Jupyter notebook was fantastic44news, because by that time the Sage Notebook's design began to feel antiquated;45moreover, Jupyter benefited from a multimillion dollar grant that resulted in46substantial open source software development and publicity.4748To address more general needs related to my teaching, I started dreaming about49creating another web application called CoCalc in 2012 (see \cite{cocalc}).50CoCalc is an abbreviation for ``Collaborative Calculation", and I hoped it could51provide a unified home for everything I was using for my teaching and research.52My favorite undergraduate course to teach at UW was called ``Mathematical53Computation"; it covered \LaTeX{}, R, Python, and computational statistics,54abstract algebra, graph theory, symbolic calculus and cryptography using SageMath, and55today it would significantly overlap with an introductory data science course. I56wanted a web application that would be ideal for hosting this course.5758Prior to launching CoCalc in 2013, I taught Mathematical Computation many times59using the Sage Notebook. However, the Sage Notebook did not adequately support60important topics that I wanted to teach, including ``How to create \LaTeX{}61documents'', ``How to use a Linux terminal'', and ``How to develop code in files62instead of a notebook''. Furthermore, it did not possess the crucial ability to63do simultaneous multiuser editing, which is a key feature of popular web64applications today. Over the years I noticed two things about my students: (1)65they appreciated working collaboratively with others, especially on final66projects, and (2) they wanted to work collaboratively with themselves via a67``time machine", in the sense that they wanted easy access to exactly what they68were doing 15 minutes ago before they messed everything up. Having mulled over69these issues, I decided I wanted to create a single web application that would70support collaborative editing of Jupyter notebooks and \LaTeX{} documents,71multiuser Linux terminals, integrated chat, and a detailed browsable time72travel history of editing documents. The first release incorporated some of73these things, but with stripped down functionality, and it was hosted on a74single desktop in my office.7576That being said, I made CoCalc public and free just like the Sage Notebook, and77a few other professors were interested in trying it out. Several students in my78course were skilled at programming and began helping with its development.79However, the instructors who first tried the application reported being annoyed80with the tedium involved with uploading and downloading assignments. To combat81this, we implemented a basic course management system, which automated82distribution and collection of assignments.8384With this course management system in place, we received much more usage.85Initially, I bought a few dozen servers using a grant, and hosted them in data86centers on the UW campus. As usage grew, I received free credits on Google87Cloud, and moved hosting to Google, which made dynamic scaling in response to88variations in load much more efficient. We ended up facing a number of89challenges due to the nature of our users, who were mainly people in courses90making heavy use of Python and R. In one course, the students would regularly91create massive plots, so we had to come up with better ways of dealing with huge92output, especially in the context of simultaneous multiuser editing. Over a93period of several years, we reimplemented the Jupyter stack for our purposes,94while attempting to preserve the functionality, look and feel of the official95Jupyter notebook.9697We also found that instructors needed access to a broad software stack, and we98accumulated hundreds of gigabytes of installed software, along with automated99scripts to install and test it. It was a major technical challenge for us to100make this software quickly available for use across the cluster, periodically upgrade it101with automated testing, and provide stable images over time. Along the way, some102of our solutions to these problems failed at scale, which resulted in lost103data, difficult weeks of hard work and many sleepless nights. Fortunately, we104eventually engineered robust solutions to these problems, which have been105working for several years.106107In 2019, I officially resigned my Full Professor position at UW to work full108time on CoCalc as a business. Today this business is reasonably stable, with a109manageable number of issues, and enough paying customers so that our team can110proactively improve CoCalc, rather than reacting to crises. The company's daily111operations are funded entirely by customers. Much of our current development112effort on CoCalc is driven by the following question, which can be asked about113every relevant piece of software: ``What would $X$ look like if one were to114focus hard for 10 years on perfecting it?" This principle guides our development115efforts, and we strive to make CoCalc faster to load and navigate, and to116improve its efficiency, stability, and usability. As an example, we are currently working to provide beginner friendly graphical user interfaces for editing117Jupyter notebooks and other documents. Most CoCalc users are beginners at118scientific computation, so a graphical editor, with an easily browsable119libraries of code snippets, makes CoCalc significantly more accessible to them.120121During the last 20 years, over one thousand people have contributed to the122online tools I have described here. Today these tools are relatively mature and123powerful. I hope you find them as useful as I do, and that maybe you will help124improve them over the coming decade.125126\begin{thebibliography}{2}127128\bibitem{cocalc} \url{https://cocalc.com}129\bibitem{emerton} \url{https://link.springer.com/article/10.1007/s00208-003-0449-2}130\bibitem{lmfdb} \url{https://www.lmfdb.org/}131\bibitem{magma} \url{http://magma.maths.usyd.edu.au/calc/}132\bibitem{sage} \url{https://sagemath.org}133134\end{thebibliography}135136137\end{document}138139140