The University of Western Ontario
London, Canada

Department of Computer Science

CS4402B/CS9635B -- Parallel and Distributed Computing

Winter 2022

Course Description

The efficient usage of parallel and distributed systems (multi-processors and computer networks) is nowadays an essential task for computer scientists.

This course studies the fundamental aspects of parallel systems and aims at providing an integrated view of the various facets of software development on such systems: hardware architectures, programming languages and models, software development tools, software engineering concepts and design patterns, performance modelling and analysis, experimenting and measuring, application to scientific computing.

Course topics may include but are not limited to: hierarchical memory, cache complexity, multi-core and many-core architectures, fork-join parallelism, scheduling, scalability, GPU computing, data parallelism, pipelining, message passing (MPI), parallel and distributed data-structures, and applications of parallel and distributed computing.

Follow this link for various resources (software tools and tutorials, hardware documentation, conferences, other HPC course web sites, etc.) regarding this course and HPC in general.


Name:Marc Moreno-Maza
Office:MC 327
Office Hours:Monday 14:30 - 15:30 or by appointment
Phone:661-2111 x3741

Lecture Notes and Textbook

Notes of each lecture will be available on the course website, approximately one or two days after the oral presentation.

The following textbooks are recommended but not required:

  1. Structured Parallel Programming by Michael McCool, Arch Robison, James Reinder.
  2. C++ Concurrency in Action, practical multithreading by Anthony Williams
  3. Programming language pragmatics (Chapter 12 only) by Michael L. Scott
  4. Models of Computation Exploring the Power of Computing by John E. Savage.

OWL Website

For both CS4402 and CS9635, the OWL web site is here.

Please check the course and OWL sites often for updates on lecture notes and errata, as well as the forum discussions and announcements.

Also be aware that the course website is not a substitute for the actual classroom attendance!

Course outline

Please find the course outline here.

Lecture Topics

The list of topics will be something on the order of:

  1. Overview of parallel and distributed computing
  2. Parallel and distributed patterns in Julia:
    1. Rough installation notes for Jupyter and Julia
    2. my css customization file for Jupyter
    3. image for blocked matrix multiplication
    4. Julia basics: functions, methods, constructors and macros
    5. Asynchronous programming and distributed programming in Julia
    6. arrays and shared arrays in Julia
  3. Hierarchical memories, cache complexity Extra materials used in the lectures:
    1. Animation of naive counting sort
    2. Animation of counting sort using buckets
    3. Animation of various transposition algorithms
    4. C programs for matrix multiplication.
  4. The fork-join model and the cilk concurrency platform Extra materials used in the lectures:
    1. The OpenCilk web site
    2. The OpenCilk Tutorial at SPAA 2020
    3. OpenCilk examples
    4. Rough installation notes for OpenCilk and the above examples.
    5. Intel Cilk++ Programmers Guide
  5. Parallel Random-Access Machines
  6. Dependence Analysis and (Automatic) Parallelization
  7. Many-core Computing with CUDA. slides (Part 1) and slides (Part 2). and simple CUDA programs and Polybench and CUDA demo.
  8. Programming parallel patterns with C++ 11 slides
  9. Multiprocessed parallelism, message passing (MPI):
    1. A Comprehensive MPI Tutorial Resource
    2. MPI intro lecture by Jon Eyolfson,
    3. MPI: Beyond the Basics by David McCaughan.
  10. MetaFork: A Compilation Framework for Concurrency Platforms Targeting Multicores slides

Class Schedule

Lectures: 3 hours (Tuesdays 12:30 - 13:30 and Thursdays 11:30 - 13:30)

When lectures are in-person, all lectures take place in TC 343.

When lectures are on-like follow this link on Tuesdays and that link on Thursdays.

Each student is expected to attend the lectures. In particular, quizzes (short written tests) may take place without notice.

Assignment/Project/Quiz Schedule

All dates are tentative and currently subject to change, although it is doubtful by any significant amount.

Evaluation Technique Weight Posted Date (tentative!) Due Date (tentative!) Workload
Assignment One1/6Jan. 27Feb. 18regular
Assignment Two1/6Feb. 18March 7regular
Project 1/3March 7See aboveheavy
Quizzes1/9 eachN/AvariousN/A

Problem Sets for 2021-2022

  • Problem set 1
  • Problem set 2
  • Quiz corrections from 2014-2015

  • Quiz 1: elements of corrections for a 2014 quiz.
  • A CUDA quiz and its elements of correction.
  • 2015 Quiz 2 with elements of corrections.
  • Quiz corrections from 2016-2017

  • Quiz 1 with elements of corrections.
  • Quiz 2 with elements of corrections.
  • CS4402 projects in 2021-2022

    Please find here the schedule of the CS4402 project presentations for the Winter 2022 ter,.

    Please find below papers that can be chosen for the course projects of CS4402.

    Marc Moreno Maza
    Last modified: Mon Jan 10, 2022