CS4472 Readings and Questions for Fall 2015

Note: quizzes 3 and 4 will be cummulative, so some of the first and second quarter material will be appearing on them.
  1. For Tuesday 20 Oct 2015 (the day of quiz 2), read the following paper Test strategies in distributed software development environments, Anuradha Mathrani and Sanjay Mathrani, Computers in Industry, Volume 64, Issue 1, January 2013, Pages 1-9 and answer the following questions:
    1. One of the concerns of this paper is the management of various risks and how they impact testing. One that they are concerned about is the exposure of confidential patient information. What is another example of a risk that they think impacts testing? Indicate the relevant paragraphs in the paper for the claim that this example is a risk that they think impacts testing.
    2. How does the beta testing approach presented in this paper relate to the lab testing and remote evaluation testing approaches of the web application's paper that was read for last Thursday? Indicate the relevant paragraphs in these papers that support your answer.
    3. Section 2 is entitled `Importance of a test strategy'. This sentiment is echoed in the lead sentence of the last paragraph of Section 4 as well. In what way do the case studies presented support this view. Indicate the relevant paragraphs or sentences of the case studies as appropriate.
  2. For Thursday, 22 Oct 2015, read Towards describing black-box testing methods as atomic rules, Murnane, T., Hall, R., Reed, K., Computer Software and Applications Conference, 2005. COMPSAC 2005. 29th Annual International (Volume:1 ), 26-28 July 2005, Page(s): 437 - 442 Vol. 2. and answer the following reaction questions:
    1. Let us go back to the marking program specification updated specification notes from quiz question 3. Let us envision a smaller specification called `simple marking task' that doesn't have reaction paper marks or project marks to deal with. What would be the equivalent of Figure 1 from the paper for the `simple marking task' specification?
    2. Say I want to generate the following test case for the `simple marking task':
      Q1: 33
      Q2: FINAL
      CurrentParticipation: 130
      Q3: 23
      Q4: 39
      FINAL: 98
      Explain how this test case could be generated by a series of application of the various rules listed in the paper to your simple marking task validation specification.
    3. This paper presents a framework of rule types. Does combinatorial testing fit into this framework?
      • a) If so, what rule type would three-way combinatorial testing be?
      • b) If not, what would need to be changed to accomodate it?
  3. For Tuesday 27 Oct 2015 Mobile Application Testing: A Tutorial, Jerry Gao, Xiaoying Bai, Wei-Tek Tsai, and Tadahiro Uehara, Computer (IEEE), Issue No.02 - Feb. (2014 vol.47), pp: 46-55.
    1. This paper (MAT) discusses testing mobile native apps and mobile web apps. In the paper `Towards an automatic evaluation of Web applications' (TAAEOWA), there was also a discussion of testing web apps in general. What in MAT comes closest to the discussion in TAAEOWA of lab testing versus remote testing? What were the differences between what MAT and TAAEOWA are saying on this topic? Indicate paragraph and/or sentences relevant to the points you raise.
    2. In order to have a complete answer to the previous question, what are all the different pieces you need to present? For the previous question did you make each of these pieces clearly identifiable so that I can tell if you included them? If not, please do so.
    3. On page 48 of MAT, 8 types of testing activities are listed. Which one or ones cover the kind of testing done in TAAEOWA? Why do you think this?
    4. In Figure 2 of MAT, why is scalability listed as an issue for Mobile Web System Testing but not for Mobile App QoS Testing (although the other four items are common to both)? Or do you believe this is a mistake on the author's part?
  4. For Thursday, 29 Oct 2015, enjoy Fall Study Break.
  5. For Tuesday, 3 Nov 2015, read Tailoring of black-box testing methods, Murnane, T., Reed, K., Hall, R., Software Engineering Conference, 2006. Australian, 18-21 April 2006, Page(s): 8 pp. - 299. and answer the following reaction questions:
    1. Syntax testing is mentioned frequently in the paper, but not actually defined. Apparently the idea originates in `Beizer B. (1990) Software Testing Techniques, Second Edition, ISBN 0-442-20672-0.' We have this book in Taylor Library (TAY stack QA76.76.T48B45 1990 -- according to Google Scholar is cited by 3,597 articles of which 96 contain the phrase `syntax testing'), but a nice overview of it can be found in Section 2.1 of Vulnerability Analysis of Software through Syntax Testing by Rauli Kaksonen (2000). At the end of the second paragraph in the second column of the first page of the Murnane article, the author says: ``An example of varying terminology can be seen ... ; in Syntax Testing (ST) [10] they are implicitly created. What has been implicitly created? Give an example based on the Foreign Currency field discussion in Section 5.
    2. In looking at the rule types for `Towards describing black-box testing ...', the question was raised as to whether or not combinatorial testing fit into this framework or not. One possible answer was that combinatorial testing was a TSSR type rule, which was `a test case construction rule'. Since combinatorial testing doesn't generate a test case, but rather generates a collection of test cases, i.e., a whole test suite, this didn't seem to be a good match. Does this second Murnane article support or oppose that view? Indicate specifically (quotes or paragraphs referenced) why you believe this.
    3. In Section 2, this paper gives us the GQAS approach for requirement elucidation. The last step of this is to `Specify the field using a formal notation'. We are then given an example of this step in the second column of Section 5 of this paper. In past papers, we have seen reference to the phrase `formal specification'. What is the difference between this specification and the way `formal specification' was used in past papers? (indicate relevant paragraphs, as usual). Is the GQAS example a `formal specification' according to these previous papers (again, indicate relevant paragraphs, as usual)?
  6. For Thursday, 5 Nov 2015 (proposal due date for people doing a 3rd quarter proposal), read: Reverse engineering and design recovery: a taxonomy (5 pages), and answer the following questions:
    1. What is the relation between `redocumentation' and `restructuring'? As usual indicate/quote the relevant parts of this paper to support your answer. Give examples to illustrate the relation.
    2. In this paper, they don't use the term `specification'. What term do they use instead? Why do you think this term corresponds to what other papers refer to as `specification'?
    3. In considering Introducing Combinatorial Testing in a Large Organization, someone said that the paper presents the results of `reverse engineering' the test data used by many Lockheed Martin projects. Is this a proper usage of the notion of `reverse engineering' as presented in the taxonomy paper? Explain, with suitable quotations from both papers.
  7. For Tuesday, 10 Nov 2015 (day of quiz 3), read: GUI ripping: reverse engineering of graphical user interfaces for testing (10 pages) and answer the following questions:
    1. Consider the web page http://dl.acm.org/author_page.cfm?id=81556203556, which we will call the Imparato page. When I type `GUI' in the search field at the upper right hand corner of the page (just under SIGN IN) and hit the search button, I end up at http://dl.acm.org/results.cfm?h=1&cfid=725996520&cftoken=63482912 which we will call the GUI search page. Ignoring all the other fields on the Imparato page, what would be the sequence of Window states that would get me from the Imparato page to the GUI search page in the manner described above according to the definitions in section 2.2? Draw each state as the node on a graph with arcs representing the transition from one state to the next. The node should be a circle inside of which is the relevant information.
    2. On the GUI search page http://dl.acm.org/results.cfm?h=1&cfid=725996520&cftoken=63482912 (note: these pages may have different names if you repeat the path I took as cfid and cftoken seem to be identifiers that the system managing these pages expires and probably reuses), A) what would be an example of a `restricted-focus event' that you could perform? B) what would be an example of an unrestricted-focus event you could perform? C) what would be an example of a termination event for this window? D) what would be an example of a menu-open event for this window? E) what would be an example of a system-interaction event for this window?
    3. In reference 17, the authors describe generating automatic GUI testing oracles from a user provided formal model of a GUI. Assuming one doesn't have such a formal model and is not interested in creating one, list all the steps you would expect to take in doing GUI testing using GUI ripping and which ones are automated and which ones need to be done manually.
  8. For Thursday, 12 Nov 2015 (due date for people doing a 3rd quarter project writeup), read: Reverse Engineering iOS Mobile Applications (10 pages) and answer the following questions:
    1. The idea of using iCrawler for `smoke testing' is mentioned a couple of times in the article. Smoke testing generally refers to quick tests that check for major problems before one settles in to do a more time consuming check (if the smoke test fails, you don't bother with the more extensive checking). What sort of simple problems could iCrawler find that would make it useful for smoke testing an iOS app?
    2. iCrawler produces a state graph for the interface to the application. In the previous paper, GUI ripping produced an event flow graph and integration tree. Aside from iCrawler being iOS specific, what is the main difference between these two processes?
    3. How does the similarlity measure usage in iCrawler relate to what GUI ripping does? Would such a measure be useful to GUI ripping? Is there something GUI ripping is doing instead of using a similarity measure?
  9. For Tuesday, 17 Nov 2015, read: Testing web applications with state objects, Arie van Deursen, Communications of the ACM, Volume 58 Issue 8, August 2015, Pages 36-43. and answer the following questions:
    1. Last full paragraph on first column of page 39 ends with `Usually, however, testing all expanded paths would be overkill for the include file setting'. How does this compare with what is going on in the GUI ripping paper? Can the GUI ripping algorithm, as defined in that paper handle this? What in that paper supports your opinion (quotes as usual).
    2. On page 42, second full paragraph on the last column, it says that if we expanded Figure 15 to make interleavings explicit, it would be a larger diagram. Please draw that expanded diagram (to simplify things a bit, remove the Sorted by C option from the Sorted Table before expanding the diagram).
    3. The only full paragraph in the middle column on page 42 contains the sentence ``The upload dialog is modal, which is why it is outside the Portfolio class.'' What is it about the dialog being modal that justifies putting it outside the class? (quotes as usual)
    4. In the Conclusion, we are told `use the transition tree to reason about state reachability and transition coverage'. Quote the first sentence (using same conventions on sentence quoting as given on Quiz 3) of each paragraph of the article that specifically relates to this point in the conclusions.
  10. For Thursday, 19 Nov 2015, [Code reviews, software inspection] read: A statistical approach to the inspection checklist formal synthesis and improvement, Chernak, Y, Software Engineering, IEEE Transactions on (Volume:22 , Issue: 12 ), Dec 1996, Page(s): 866 - 874 and answer the following questions:
  11. For Tuesday, 24 Nov 2015, [Cloud testing] read: Framework for testing cloud platforms and infrastructures, Jenkins, W., Vilkomir, S., Sharma, P., and Pirocanac, G., Cloud and Service Computing (CSC), 2011 International Conference on, 12-14 Dec. 2011, Page(s): 134 - 140. and answer the following questions:
    1. More most of the testing we have looked at, you have a system under test and you apply a series of tests in sequence to it to determine whether or not it works. However, in testing a system like the Google App APIs, there is the possibility of running different tests against different API functions at the same time. The question is, is this what they are actually doing in the process discussed in this paper? State what you think the answer is and then quote the sentences (from this paper) relevant to the matter and in particular, your position on the matter.
    2. In the conclusions, the authors say `we can also expand upon automatic oracle generation'. In the work reported in the paper, how are testing oracles handled? Again, quote the relevant sentences from this paper on the matter.
    3. On page 136, 2nd column, 2nd paragraph, the authors say it is unrealistic to run 768 tests. The tests look rather simple. Why do they take so much more time than tests of the party-sample.c program (from class)?
  12. For Thursday, 26 Nov 2015, [Software reverse engineering] read: Theories, methods and tools in program comprehension: past, present and future, Storey, M., Program Comprehension, 2005. IWPC 2005. Proceedings. 13th International Workshop on, 15-16 May 2005, Page(s): 181 - 191. and answer the following questions:
    1. Referring to the program party-sample_records.c, which we have discussed in class (and will be added to the material that may be included with the printouts that are referred to by the final exam questions),
      1. What would be an example of a programming plan illustrated by this code? Describe briefly the plan and indicate the code line numbers.
      2. What would be an example of a beacon from this program. What are its features? What structure is it a cue for?
      3. A `delocalized plan' is mentioned in the paper a couple of times, but not actually defined. The idea is that the `plan' corresponds to a decision that is not localized in the code to a particular range, but is instead `scattered' across different parts of the code. For example, in the mark151014.c program, the decision to use the projectSkip array to track the occurance of the value SKIP for a project is a plan one only understands by looking at its usage across main, readData, readProject, computeScore, projectScore, and bonusScore. Bits of code scattered throughout these functions reflect the decision to handle SKIP this way rather than storing a value of -1 to indicate SKIP as is done with representing the value FINAL in the context of quizzes. What would be an example of a delocalized plan in the party-sample_records.c program?
    2. The other word that is significant, mentioned in this paper, but not defined, is `slicing techniques' (see paragraph that starts Biggerstaff, which is the second paragraph of Section 3.2). This term is discussed in more detail on wikipedia page on Program slicing. There we are given a rather formal definition as well as some examples (the definition is:
      • Based on the original definition of Weiser, informally, a static program slice S consists of all statements in program P that may affect the value of variable v at some point p. The slice is defined for a slicing criterion C=(x,V), where x is a statement in program P and V is a subset of variables in P. A static slice includes all the statements that affect variable v for a set of all possible inputs at the point of interest (i.e., at the statement x). Static slices are computed by finding consecutive sets of indirectly relevant statements, according to data and control dependencies.
      Clearly that sentence would be a bit out of place in this paper. On the other hand, the author does introduce technical terms more casually, for example the phrase `concept assignment problem' that also appears in the Biggerstaff paragraph cited above. How could the sentence including `slicing techniques' in that paragraph have been expanded/rewritten a little (a few words) to convey the intuitive meaning being referred to? Give your new improved version of that sentence.
  13. For Tuesday 1 Dec 2015 (day of quiz 4 and proposal for fourth quarter second default project due), [Regression testing] read: Automated Behavioral Regression Testing, Wei Jin, Orso, A., and Tao Xie, Software Testing, Verification and Validation (ICST), 2010 Third International Conference on, 6-10 April 2010, Page(s): 137 - 146. and answer the following reaction questions:
    1. On page 139, at the end of the next to last paragraph of Section II, it says that `In the case of object-oriented languages, the oracle problem is further complicated by the presence of ... information hiding'. Why would information hiding in an object-oriented language be a problem for testing and the checking of test results?
    2. Normally, a JUnit test wouldn't be able to access the isOverdraft field in Figure 2 since it is declared private. Is it visible to BERT? Quote the relevant sentences in the paper that support your answer.
    3. BERT runs test cases for classes that have been changed. Does it run only on methods that have been changed or does it run on some methods that have not themselves been changed? Quote the relevant sentences in the paper that support your answer.
  14. For Thursday, 3 Dec 2015, [Experiments evaluating testing techniques] read: How do professional developers comprehend software?, Roehm, T., Tiarks, R., Koschke, R., and Maalej, W., Software Engineering (ICSE), 2012 34th International Conference on, 2-9 June 2012, Page(s): 255 - 265. and answer the following questions:
    1. One study of duplicate code in programs found that 8.7% of the GCC compiler (written in C) contained duplicated sequences of code (ignoring white space and comments) of 10 or more lines. A similar statistic for a message board program written in Python was 29.4%. What does Rohm et al say about why this happens (A) in your own words and (B) quoting the relevant sentences from the paper.
    2. Today in class, we looked at Cucumber in more detail ( wikipedia entry for the software testing package). Which hypothesis (hypotheses) would using Cucumber properly help with? Why do you think it is relevant to these issue(s)?
    3. One problem with studies like this is that sometimes the biases of the person doing the observations impacts what is observed. How do the authors of this paper address this problem (your words)? What is the relevant quote from the paper?
  15. For Tuesday, 8 Dec 2015 (last day of class and writeup for fourth quarter second default project due), [Web applications] read: Bypass testing of Web applications, Offutt, J., Wu, Ye., Du, X., and Huang, H., Software Reliability Engineering, 2004. ISSRE 2004. 15th International Symposium on, Page(s): 187 - 197. and answer the following reaction questions, the last of the semester:
    1. How does the DFS to identify input patters of Web applications differ from GUI ripping? Specifically,
      • A) Do they traverse a web application in the same way, or does one visit some parts of an application that the other wouldn't?
      • B) Do they both return the same information to the user? If not, what information is unique to each technique.
    2. On the page that contains the start of Section 3, in the second paragraph of the first column, the authors say they `broke the inputs into three levels ... this process revealed five types of faults'. List the five faults (numbered 1 to 5 in that column) and indicate which level of input they are associated with.
    3. According to this paper, how is the buffer overflow problem related to the bypassing validation problem?
nostalgia (links to past year's material that may be brought into this year's readings):