++++++++++++++++++ FME2001: - Review reports for paper 57 +++++++++++++++++++++ - Dear author(s) : this file was extracted automatically from a very large - mailbox. In case you find any problem concerning mail encodings (or any - other kind of anomaly disturbing your understanding of the reviews) please - email any of the PC cochairs (pamela@research.att.com,jno@di.uminho.pt). ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ PAPER NUMBER: 57 CATEGORY: 2 TITLE: Metrics for Test Plan on Object-Oriented Behavioral Design AUTHOR(S): Young Chul Robert Kim C. Robert Carlson -------------------------------------------------------------------------------- 1. Briefly SUMMARIZE the paper (2-3 lines): Metrics for test plans based on interaction diagrams are given. 2. RELEVANCE: Please provide a rating of the paper's relevance to the FME Symposium, using the scale: 0 = Out of scope 1 = Marginal interest 2 = Minority interest 3 = Majority interest 4 = Outstanding interest Numeric Rating: 0 Please comment your rating: The paper is not about formal methods, but about semi-automatic analysis of quasi-formal design methods. 3. OVERALL RECOMMENDATION: Please provide a rating of the paper's acceptability, using the scale: 1 = Strong reject 2 = Weak reject 3 = Could go either way 4 = Weak accept 5 = Strong accept Numeric Rating: 1 NB: There should be a correlation between the two rates above. 4. CONFIDENCE LEVEL: Please provide a rating oof your expertise in the area addressed by the paper, using the scale: 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average I'd say 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it extremely well Numeric Rating: 2 NB: PC members are responsible for ensuring that 1 is not used here. 5. ORIGINALITY. What is NEW and SIGNIFICANT in the work reported here? Comment: Hard to say. Lots of references to the use-case world, which I don't know well enough to say what's new here. Some of the "theory" in this paper is trivial, e.g. the diagram on p3 to illustrate all 4 possible combinations of multiple/single input/output. 6. How WORTHWILE is the goal of the authors? Comment: Sensible enough: to develop test plans in design stage, and to select for reusability in those. 7. How well is this goal EXPLAINED and JUSTIFIED? Comment: Reasonably. 8. TECHNICAL QUALITY. Are the technical parts (definitions, statements, specifications, proofs, algorithms, etc.) SOUND? Comment: I think so, but also trivial. 9. APPLICABILITY. If the work is primarily theoretical or conceptual, are the implications in terms of applicability adequately justified? If the paper is about a new formal technique, are satisfactory arguments presented in favor of the new technique? If a methodology is proposed, is it sound? If experimental results are presented, is their relevance justified? Comment: Viewing this as conceptual work, the implications in terms of applicability are hardly mentioned. 10. PRESENTATION: Describe the QUALITY of the writing, including suggestions for changes where appropriate. Comment: Improvements necessary in every second sentence: missing words, strange word order, etc. Structure unclear at every level, too much jargon. Too many diagrams of too many different kinds break up the coherence of the paper. 11. Were there any formatting or mechanical problems with this paper?: No. Are the figures and length acceptable?: Yes. Are the references correct?: I believe so. 12. OTHER COMMENTS you believe would be useful to the author(s), including pointers to missing relevant work: +++++++++++++++++++++ End of FME 2001 Paper Review Report ++++++++++++++++++++++ PAPER NUMBER: 57 CATEGORY: 2 TITLE: Metrics for Test Plan on Object-Oriented Behavioral Design AUTHOR(S): Young Chul Robert Kim C. Robert Carlson -------------------------------------------------------------------------------- 1. Briefly SUMMARIZE the paper (2-3 lines): This paper is about deriving test plans when a use case methodology is used. It is purely a UML paper without any formal contents whatsoever. 2. RELEVANCE: Please provide a rating of the paper's relevance to the FME Symposium, using the scale: 0 = Out of scope 1 = Marginal interest 2 = Minority interest 3 = Majority interest 4 = Outstanding interest Numeric Rating: 0 Please comment your rating: Should not be necessary :-) 3. OVERALL RECOMMENDATION: Please provide a rating of the paper's acceptability, using the scale: 1 = Strong reject 2 = Weak reject 3 = Could go either way 4 = Weak accept 5 = Strong accept Numeric Rating: 1 NB: There should be a correlation between the two rates above. 4. CONFIDENCE LEVEL: Please provide a rating oof your expertise in the area addressed by the paper, using the scale: 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average I'd say 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it extremely well Numeric Rating: 4 NB: PC members are responsible for ensuring that 1 is not used here. 5. ORIGINALITY. What is NEW and SIGNIFICANT in the work reported here? Comment:Nothing 6. How WORTHWILE is the goal of the authors? Comment: Poor 7. How well is this goal EXPLAINED and JUSTIFIED? Comment: Poor 8. TECHNICAL QUALITY. Are the technical parts (definitions, statements, specifications, proofs, algorithms, etc.) SOUND? Comment: Impossible to say 9. APPLICABILITY. If the work is primarily theoretical or conceptual, are the implications in terms of applicability adequately justified? If the paper is about a new formal technique, are satisfactory arguments presented in favor of the new technique? If a methodology is proposed, is it sound? If experimental results are presented, is their relevance justified? Comment: No 10. PRESENTATION: Describe the QUALITY of the writing, including suggestions for changes where appropriate. Comment: Extremely informal and algorithmic 11. Were there any formatting or mechanical problems with this paper?: No Are the figures and length acceptable?: Yes Are the references correct?: Yes 12. OTHER COMMENTS you believe would be useful to the author(s), including pointers to missing relevant work: I think that this paper has been submitted to the wrong conference. In addition it would be recommendable to get a native speaker reading it to get rid of the worst mistake in the English formulation. +++++++++++++++++++++ End of FME 2001 Paper Review Report ++++++++++++++++++++++ PAPER NUMBER: 57 CATEGORY: 2 TITLE: Metrics for Test Plan on Object-Oriented Behavioral Design AUTHOR(S): Young Chul Robert Kim C. Robert Carlson -------------------------------------------------------------------------------- 1. Briefly SUMMARIZE the paper (2-3 lines): The approach described in this paper starts from sequence diagrams developed in the design phase to specify use case scenarios. These diagrams are partitioned into sequences of design units. The authors introduce a number of different partition strategies and describe a conversion algorithm from sequence diagrams to design units. A test plan consists of an ordered set of test scenarios obtained by partioning the sequence diagrams.The authors propose a number of metrics by which a priority ordering on these test scenarios can be established. 2. RELEVANCE: Please provide a rating of the paper's relevance to the FME Symposium, using the scale: 0 = Out of scope 1 = Marginal interest 2 = Minority interest 3 = Majority interest 4 = Outstanding interest Numeric Rating: 2 Please comment your rating: The topic of the paper (scenario- based testing) is interesting for the conference. The paper, however, is not written in a clear enough way to be ripe for publication. Moreover, the paper is only about ordering test scenarios, but does not really introduce a testing method. The authors introduce their testing approach as an approach "which emphasizes testing software design specifications in the design phase", but except for a short introduction of testable state pairs it is not worked out properly how design specifications are actually tested and what the criteria for a sucessful or failed test are. 3. OVERALL RECOMMENDATION: Please provide a rating of the paper's acceptability, using the scale: 1 = Strong reject 2 = Weak reject 3 = Could go either way 4 = Weak accept 5 = Strong accept Numeric Rating: 2 NB: There should be a correlation between the two rates above. 4. CONFIDENCE LEVEL: Please provide a rating oof your expertise in the area addressed by the paper, using the scale: 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average I'd say 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it extremely well Numeric Rating: 3 NB: PC members are responsible for ensuring that 1 is not used here. 5. ORIGINALITY. What is NEW and SIGNIFICANT in the work reported here? Comment: What seems to be new is that Hulburt's action matrix concept is enhanced in order to be used for scenario-based testing. The metrics for ordering test scenarios have been partly obtained by adaptation of Musa's work. The approach in this paper differs in that reusability aspects are also incorporated into the metric. 6. How WORTHWILE is the goal of the authors? Comment: The goal seems to be an approach to test design specifications which is a worthwhile goal. 7. How well is this goal EXPLAINED and JUSTIFIED? Comment: The goal in itself is hardly explained. As stated in 2. very little is said about the actual testing approach. The focus in this paper is on the partitioning of sequence diagrams into design units and on the metrics used on test scenarios. The conversion algorithm from sequence diagrams to design units is presented in a way which is not understandable to the referee. Neither does it come out in 5.1 (Test Plan Metrics) how exactly the metrics of "most reusable subpaths" is defined. 8. TECHNICAL QUALITY. Are the technical parts (definitions, statements, specifications, proofs, algorithms, etc.) SOUND? Comment: - the conversion algorithm in chapter 4 is not presented in a formally acceptable way (see also 12c)). - the definition of the "total probability of occurrence of a use case scenario" lacks some important details: what is probability i ? And is it really the probability of action unit i what is needed in the formula, and not rather the probability of a link in the Mealy model (as in the example in Figure 10)? - the definition of the "Collection of Total Reusable Weight Values and Total Critical Weight Values" (page 18) remains utterly unclear. 9. APPLICABILITY. If the work is primarily theoretical or conceptual, are the implications in terms of applicability adequately justified? If the paper is about a new formal technique, are satisfactory arguments presented in favor of the new technique? If a methodology is proposed, is it sound? If experimental results are presented, is their relevance justified? Comment: It is not mentioned in the paper on which examples or in which projects the method has been applied yet. The example presented in the paper is not the design of a software application, but rather the decription of a business process involving manual activities- therefore it is not a convincing application. 10. PRESENTATION: Describe the QUALITY of the writing, including suggestions for changes where appropriate. Comment: - the English must be improved, since there are many grammatical errors which in some places make the paper hard to read. - since there are three authors, they should not refer to themselves in the singular (as I...). 11. Were there any formatting or mechanical problems with this paper?: no Are the figures and length acceptable?: yes Are the references correct?: there is a wrong reference ([Carl99]) on page 4. 12. OTHER COMMENTS you believe would be useful to the author(s), including pointers to missing relevant work: a) The restaurant example used in the paper is not an example of a software system, but rather of a business process involving manual activities (like cooking). Therefore it is not a good example for software testing. b) The header "A layered design unit architecture" of chapter 2 is not adequate, since it is never mentioned in which way the design units are layered. Moreover, the term "architecture" should not be used here, since a global system architecture is never mentioned. c) Figure 11a) (The main structural chart of the conversion alg.): What do you mean by "Its message arrives Actor ?" And how is it determined whether the outcome of Decision ? is positive or negative ? A different way of presenting the algorithm should be chosen. The charts involving goto's are hardly understandable. d) In Figure 10) the sum of the probabilities of the outgoing links of a node are always 1 except for the first node, where the sum is 0.9. There should be an explanation given for this (The links are not yet complete ?). In Figure 10a) above, in the second row (Variant1) the number of the total probability of occurrences is miscalculated (0.2673 is the right number - as in Figure 13a). e) Section 2.1 Definition of Design Units, page 4, paragraph with the header State Unit, third line: "Figure 4" is missing. +++++++++++++++++++++ End of FME 2001 Paper Review Report ++++++++++++++++++++++