CS614B - Advanced Topics in Software Quality

Course Outline - January 2006

Professor: Mechelle Gittens PhD.

More than a third of all costs in software development come from finding, preventing and removing defects before the software is delivered. In addition, organizations that give their quality processes low priority get stuck in high service and maintenance costs after software release. There are several stories of calamities that occurred because a company paid little attention to quality control and assurance. The National Institute of Standards and Technology (NIST) tells us that software errors cost the U.S. economy about $59.5 billion a year.

There has been significant work done in software engineering to address these issues, and in this course, we will investigate many of these proven and emerging methods.

Welcome to Advanced Topics in Software Quality!

Announcements

Assignment 3 - Submissions

Documentation: Please document all code, so that the logic can be followed. Include file headers, and reasonable inline documentation.
Implementation of the functionality: Please submit all code for the system specified. Submit all supporting scripts, programs and files including the code to generate your own data and the samples of the data files (if applicable).
Test output: Please include some sample of the output that satisfies the three kinds of testing required by the assignment: unit testing, integration testing and system testing. You may include this sample when you discuss how your team approached the required testing. Also, include a narrative or description that outlines how to navigate your program to satisfy each of the scenarios.

Click here for your peer evaluation form. Please complete and bring this form to class on April 6th.

Guest Talk . April 6th - 1pm: Dr. Matunda Nyanchama (IBM)

Matunda Nyanchama PhD. CISSP is the Canadian Delivery Project Executive for Security, Identity and Privacy Services. Previously, he was in charge of service delivery in the Security, Identity & Privacy Practice. In the past, he has worked as a Senior Manager of Information Security and Risk Management at Moneris Solutions, a payment solutions company based in Toronto, Canada. Prior to that he was a Senior Advisor for Information Security Analytics at the Bank of Montreal Financial Group, where he focused on information security risk analytics, business strategy and security awareness. Dr Nyanchama has held a number of professional security positions, including Senior Security Consultant at Ernst & Young and Director of Security Architecture, Intellitactics Inc., a Canadian security software company. Matunda holds masters and doctoral degrees in computer science from the University of Western Ontario in Canada, and an undergraduate electrical engineering degree from the University of Nairobi in Kenya. He is a certified information systems security professional (CISSP). He has presented on the subject of Sarbanes-Oxley compliance and security and written about information security metrics. Dr Nyanchama has published a number of security management papers, including co-authoring a chapter in the Information Security Management Handbook the reference guide for CISSP. His doctoral work in Role-Based Access Control (RBAC) is widely cited in computer security literature.

Abstract: Security breaches make headlines on an ongoing basis while companies lose valuable time and incur losses in responding to security incidents following the exploitation of software flaws. According to the 2005 CSI/FBI Computer Crime and Security Survey, viruses, worms and other malware continue to cause substantial losses in industry. Some think that the reported losses constitute a tip of the iceberg of cumulative loss. Risks associated with flaws in software could be much larger. Security flaws have origins in how security software is developed. The talk will focus on the need for a disciplined approach to software development and the adoption of engineering practices as a basis for ensuring secure software.

Guest Talk . March 23rd: Test Automation Tooling - Anthony Di Loreto, B.Eng, P.Eng. (IBM)

Anthony Di Loreto graduated with a degree in Computer Engineering from McMaster University in 1998 . Before coming to work for IBM in 2000, Anthony worked at an engineering firm in database development. This helped start his career in Database Technology at IBM, where he has worked for over six years in the System Verification Testing area of DB2. In SVT, Anthony has tested DB2, written test scenarios, and has focused efforts in authoring and maintaining automation test tools. Currently, he is the test technical lead for the XML technology in the latest version of DB2. Anthony has received recognition for his work in testing and test tool automation and has two test patents pending.

Abstract: As software products grow larger and more sophisticated, so too must the automated test tools. We can no longer treat the testing of a product with thousands or millions of lines of code with simple scripting. Test automation touches many areas of a software product, including performance, regression, integration, configuration, and stress testing. To achieve results, companies must realize there is a large up front cost to automation testing, and the payback comes later when teams adopt the processes and tools. A case study of test automation is explored in the area of System Verification Testing. Ideas about creating and using automation test tools are described, as are personal experiences and recommendations. Automation tooling can never replace a tester. Tooling simply does what it is programmed to do. The creativity required to come up with effective test ideas, combinations of such ideas, and the creation of tools to encapsulate them must still come from the human expert with domain knowledge. Click here for the slides, and here for the testing example.

March 1/2006: Update to Assignment 3 made on page 2. The URL for The Canada Post Postal Code Lookup website has been corrected. It is "http://www.canadapost.ca/tools/pcl/bin/advanced-e.asp"

Guest Talk . February 23rd: An Overview of Software Estimation Models - Danny Ho, M.Sc., P.Eng., PMP (Motorola)

Danny Ho works as the Manager of Engineering Program in Motorola Canada Limited. Prior to joining Motorola, he held management and senior technical positions at Nortel Networks Corporation and IBM Canada Limited. He is also appointed as an Adjunct Research Professor at the Faculty of Engineering, the University of Western Ontario. Throughout his professional career, he has led programs in the areas of wireline, RF, and infrared development, desktop application deployment, reuse, and software development environment. His areas of special interest include software estimation, project management, object-oriented software development, and complexity analysis.

Danny received his Honors Bachelor of Science in Computer Science with Electrical Engineering, and Master of Science in Computer Science from the University of Western Ontario. He is currently a member of the Professional Engineers Ontario (PEO) and a Project Management Professional (PMP).

Abstract: Today, when software development has become an essential investment for many organizations, software engineering practitioners become more and more concerned about accurately predicting the cost and quality of a software product under development. Estimation accuracy is largely affected by modeling accuracy. Finding good models for software estimation is now one of the most important objectives of the software engineering community.

Some difficulties with software estimation are dealing with imprecise and uncertain information, and highly complex nonlinear relationships between variables. There are promising techniques such as fuzzy logic, artificial neural networks, and evolutionary computation for software modeling. The neuro-fuzzy approach, by symbiotically integrating the merits of fuzzy logic and neural networks, provides a more powerful tool to solve many issues in software estimation. Such a model can be easily interpreted and generalized, and has learning and adaptation capability.

This tutorial will provide an overview of software estimation models, and an in-depth discussion on the theory and experience in using the Constructive Cost Model (COCOMO) and Software Life Cycle Model (SLIM). The tutorial also highlights the Neuro-Fuzzy Algorithmic (NFA) Model that combines neural network, fuzzy logic and an algorithmic estimation model. The experience and result, with industrial project data, will be presented in making use of NFA for software cost estimation. Three models namely the COCOMO Model, Analysis of Variance (ANOVA), and Function Point Analysis are used as examples of algorithmic models. The tutorial concludes with the roadmap and direction to enrich the estimation models in tackling different estimation challenges.Click here for slides.

Guest Talk . February 9th: End to End Software Quality . Mr. Mark Wilding (IBM)

Mark Wilding is a senior developer at IBM who specializes in Quality and Serviceability. With over 20 years of experience writing software, Mark has extensive expertise in operating systems, networks, C/C++ development, and computer hardware.

Abstract: This presentation will cover the complete spectrum of Quality from user requirements to statistical analysis of customer problems (and everything in between). The session will be filled with industry experience as well as lessons from society's struggle with software quality. Click here for slides.

Prerequisites:

Undergraduate software engineering, a programming language such as C/C++ or Java, and basic statistics are required.

Course Evaluation:

Your performance in the course will be evaluated by all of the following methods: one (1) individual written assignment (15%) ; one (1) individual project & presentation (30%) ; one (1) group project and presentation (40%); and through class participation (15%).

Class Format:

Participation is the key to your success. The topics for the course will be covered through lectures, in-class discussions, guest lectures, student presentations, assigned readings, projects, and assignments.

Time: 9:30 am - 12:00 noon and 1:00 pm - 3:30 pm

Dates: January 12, 26; February 9, 23; March 9, 23; April 6

Day: Thursday

Office Hours: Thursday 3:30pm – 4:30pm

Contact: mgittens@ca.ibm.com

Plagiarism:

The UWO Senate Academic Handbook has specified that the following points should be added to all course outlines:

1. Plagiarism: Students must write their essays and assignments in their own words. Whenever students take an idea, or a passage from another author, they must acknowledge their debt both by using quotation marks where appropriate and by proper referencing such as footnotes or citations. Plagiarism is a major academic offence (see Scholastic Offence Policy in the Western Academic Calendar).

2. Plagiarism Checking: The University of Western Ontario uses software for plagiarism checking. Students may be required to submit their written work and programs in electronic form for plagiarism checking.

3. Prerequisites for a course: Unless you have either the requisites for this course or written special permission from your Dean to enroll in it, you will be removed from this course and it will be deleted from your record. This decision may not be appealed. You will receive no adjustment to your fees in the event that you are dropped from a course for failing to have the necessary prerequisites.

4. If computer-marked multiple-choice tests and/or exams are given: Use may be made of software to check for unusual coincidences in answer patterns that may indicate cheating.

Below is an outline of the topics to be covered in this course. This is however a guide since a few of the topics may change according to the availability of speakers. Additional topics may also be covered.

The slides for the course will be added beside the relevant date as the course progresses.

Outline:

January 12: Introduction & Quality Management

· Introduction to this class

· What is software quality?

· Standards

· Quality methodologies and processes

Lecture 1 Notes

January 26: Software Engineering and Information Management

· Life cycles and architectures

· Requirements Management

· Requirements Engineering

· Maintenance management

· Configuration Management

· Quality management

· Individual written assignment out

Lecture 2 Notes

· Readings for this class:

· A systematic survey of CMM experience and results. James D. Herbsleb, Dennis R. Goldenson May 1996 Proceedings of the 18th international conference on Software engineering

· What small business and small organizations say about the CMM: experience report. Judith G. Brodman, Donna L. Johnson May 1994 Proceedings of the 16th international conference on Software engineering

· A survey of industrial experiences with CMM and the teaching of CMM practices. Erol Biberoglu, Hisham Haddad December 2002 Journal of Computing Sciences in Colleges, Volume 18 Issue 2

· Requirements engineering: Crosscutting quality attributes for requirements engineering. Ana Moreira, João Araújo, Isabel Brito. July 2002. Proceedings of the 14th international conference on Software engineering and knowledge engineering (SEKE '02). ACM Press.

· Measuring requirements testing:: experience report. Theodore Hammer, Linda Rosenberg, Lenore Huffman, Lawrence Hyatt. May 1997. Proceedings of the 19th international conference on Software engineering. ACM Press.

· Software Quality Development and Assurance in RUP, MSF and XP - A Comparative Study. Wolfgang Zuser, Stefan Heil, Thomas Grechenig. May 2005. ACM SIGSOFT Software Engineering Notes , Proceedings of the third workshop on Software quality 3-WoSQ, Volume 30 Issue 4. ACM Press.

· Customer Relationships and Extreme Programming. Paul S Grisham, Dewayne E. Perry. May 2005. ACM SIGSOFT Software Engineering Notes , Proceedings of the 2005 workshop on Human and social factors of software engineering HSSE '05, Volume 30 Issue 4. ACM Press.

February 9: Testing vs. Quality

· Is there a difference between testing and quality assurance?

· Testing levels and testing techniques

· Testing processes

· Testing measures

· The cost of quality

· Individual written assignment due

· Individual projects out

Lecture 3 Notes

· Quality engineering: Cost, quality and user satisfaction of software products: an empirical analysis. M. S. Krishnan. October 1993. Proceedings of the 1993 conference of the Centre for Advanced Studies on Collaborative research: software engineering - Volume 1. IBM Press.

· Evaluating the cost of software quality Sandra A. Slaughter, Donald E. Harter, Mayuram S. Krishnan. August 1998. Communications of the ACM, Volume 41 Issue 8. ACM Press.

· Q focus: quality assurance: Quality assurance: much more than testing Stuart Feldman. February 2005. Queue, Volume 3 Issue 1. ACM Press.

February 23: White-box vs. Black-box testing

· Control flow testing

· Data flow testing

· Integration testing

· Equivalence partitioning

· Complexity Analysis for testing

· Individual projects due

· Individual student presentations

· Group projects out

Student Individual Presentations: February 23 . 1pm

1. Mike Hourahine

2. Janette Wong

3. Stephen Shanwei Xu

4. Belal Tassi

Readings for this class:

ACM Portal

· Software engineering environment: Software estimation using the SLIM tool. Nikki Panlilio-Yap. November 1992. Proceedings of the 1992 Conference of the Centre for Advanced Studies on Collaborative research. IBM Press.

· An assessment and comparison of common software cost estimation modeling techniques. Lionel C. Briand, Khaled El Emam, Dagmar Surmann, Isabella Wieczorek, Katrina D. Maxwell. May 1999. Proceedings of the 21st international conference on Software engineering. IEEE Computer Society Press

· The business of software: Ten unmyths of project estimation. Phillip Armour. November 2002. Communications of the ACM, Volume 45 Issue 11. ACM Press.

(also available from http://www.loai-naamani.com/Academics/Concepts/armour.pdf)

IEEE Portal

· A Neuro-Fuzzy Model for Software Cost Estimation. Xishi Huang, Luiz F. Capretz, Jing Ren, Danny Ho. QSIC, p. 126, Third International Conference On Quality Software, 2003. IEEE Press.

March 9: Testing Object-Oriented systems and Web-based applications

· Class testing

· Integration testing

· System Testing

· Functional testing

· Test Automation

· Individual student presentations

Student Individual Presentations: March 9 . 1pm

5. Edwin Chan

6. Rajan Bhakta

7. Yi Luo

8. Lawrence Mandel

Lecture 5 Notes

Readings for this class:

· Developing an object-oriented software testing and maintenance environment. David Kung, Jerry Gao, Pei Hsia, Yasufumi Toyoshima, Chris Chen, Young-Si Kim, Young-Kee Song. October 1995. Communications of the ACM, Volume 38 Issue 10. ACM Press.

· Automated Testing of Classes. Ugo Buy, Alessandro Orso, Mauro Pezze. August 2000. ACM SIGSOFT Software Engineering Notes , Proceedings of the 2000 ACM SIGSOFT international symposium on Software testing and analysis ISSTA '00, Volume 25 Issue 5. ACM Press.

March 23: Defect Prediction

· Preventing defects: Defect root cause analysis

· Assessing V&V effectiveness: Orthogonal defect Classification

· Defect content models

· Measuring and predicting software reliability

· Certifying software reliability

· Software Estimation

Lecture 6 Notes

Readings for this class (review a previous reading on automation) :

April 6: Project Presentations

· Group presentations

*Group 1*	Yi Luo, Shanwei Stephen Xu, Mike Hourahine, Lawrence Mandel
*Group 2*	Janette Wong, Edwin Chan, Rajan Bhakta, Belal Tassi

· Group projects due

· Security and Software Quality

Assignments:

Assignment 1 - Due February 9th: To get the article distributed in class click here.
Assignment 2 - Due February 23rd
Assignment 3 - Due April 6th in class - Zipped electronic and paper copies.

Reference Materials:

The texts are not required for the course, since the course slides will provide the content for the course. However they are valuable background for the materials covered in the course. The papers will be assigned as readings for course discussion, and will be added to the list as required. They can be accessed through the university library portals.

Texts:

1. Highly recommended: Stephen H. Kan. Metrics and models in software quality engineering, 2^nd edition. Addison-Wesley, 2003.

2. Edward Kit. Software Testing In the Real World: Improving the Process. ACM Press, 1995.

3. William E. Perry. Effective Methods for Software Testing. John Wiley and Sons, Inc., 2 edition, 2000.

4. Brian Marick. The Craft of Software Testing: Subsystem Testing including object-based and object-oriented testing. Prentice Hall PTR. 1995

5. Norman E. Fenton, Shari Lawrence Pfleeger. Software Metrics. PWS Publishing Company. 1997

6. Martin Wieczorek, Dirk Meyerhoff. Software Quality. Springer. 2001

7. G. Gordon Schulmeyer, James McManus. The Handbook of Software Quality Assurance, 3^rd Edition. Prentice Hall PTR. 1999

8. John Musa. Software Reliability Engineering. McGraw-Hill - 1999.

9. Boehm, Barry W., Englewood Cliffs: Software Engineering Economics. Prentice Hall, 1981.

10. Michael R. Lyu (Editor). Handbook of Software Reliability Engineering. IEEE Computer Press - 1995.

Papers:

1. Norman E. Fenton, Nicolae Ohlsson. Quantitative Analysis of Faults in a Complex Software System, IEEE Transactions on Software - 2000. Vol. 26 No. 8 (The Pareto Principle )

2. Edward N Adams. Optimizing Preventive Service of Software Products, IBM Journal of Research - 1984. Vol. 28 No. 1

3. L. Bernstein. Tidbits, ACM SIGSOFT - Software Engineering Notes - 1993 Vol. 18 No. 3

4. Antonia Bertolino (Editor). The Software Engineering Body of Knowledge - Testing. IEEE and SWEBOK Consortium - 2000. www.swebok.org

5. CMU SEI. A Framework for Software Product Line Practice - Testing. 2001 - January. www.sei.cmu.edu/productlines/frame_report/testing.htm

6. Alberto Avritzer and Elaine J. Weyuker. Preventive service of software products. IBM Journal of Research, 28(1), January 1984.

7. Barry Boehm. Software engineering. IEEE Transactions on Computers, C-25, 1976.

8. D. W. Carman, A. A. Dolinsky, M. R. Lyu, , and J. S. Yu. Software reliability engineering study of a large-scale telecommunications software system. In Proceedings of the 6th International Symposium on Software Reliability Engineering. IEEE, 1995.

9. Mechelle Gittens, Hanan Lutfiyya, David Godwin, Michael Bauer, Yong Woo Kim, and Pramod Gupta. An empirical evaluation of system and regression testing. In Proceedings of CASCON 2002. IBM, 2002.

10. Mechelle Gittens, Yong Woo Kim and David Godwin. The Vital Few versus the Trivial Many: Examining the Pareto Principle for Software. In Proceedings of the Twenty-Ninth International Computer Software and Applications Conference. Edinburgh, Scotland. July 2005.

11. Telcordia. Telcordia software visualization and analysis toolsuite (xsuds). http://xsuds.argreenhouse.com/html-man/, July 1998.

12. Jeffrey Voas. Certifying software for high-assurance environments. IEEE Software,

13. 16(4), July/August 1999.

14. Elaine J. Weyuker and Fillipos I. Vokolos. Experience with performance testing of software systems: Issues, approach, and case study. IEEE Transactions on Software Engineering, 26(12), December 2000.

15. James A. Whittaker and Jeffrey Voas. Toward a more reliable theory of software reliability. IEEE Computer, 33(12), December 2000.

Further references will be added.