++++++++++++++++++ FME2001: - Review reports for paper 05 +++++++++++++++++++++ - Dear author(s) : this file was extracted automatically from a very large - mailbox. In case you find any problem concerning mail encodings (or any - other kind of anomaly disturbing your understanding of the reviews) please - email any of the PC cochairs (pamela@research.att.com,jno@di.uminho.pt). ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ PAPER NUMBER: 05 CATEGORY: 2 TITLE: Towards execution in automatic test suite generation AUTHOR(S): Yixin Zhao Jianping Wu -------------------------------------------------------------------------------- 1. Briefly SUMMARIZE the paper (2-3 lines): The paper is about an improvement of a system for automatically generating a test suite called TUGEN. 2. RELEVANCE: Please provide a rating of the paper's relevance to the FME Symposium, using the scale: 0 = Out of scope 1 = Marginal interest 2 = Minority interest 3 = Majority interest 4 = Outstanding interest Numeric Rating: 0 Please comment your rating: The only thing this paper has of relevance for FME is the existence of a normal form for representing predicates. Pure rubbish! 3. OVERALL RECOMMENDATION: Please provide a rating of the paper's acceptability, using the scale: 1 = Strong reject 2 = Weak reject 3 = Could go either way 4 = Weak accept 5 = Strong accept Numeric Rating: 1 NB: There should be a correlation between the two rates above. 4. CONFIDENCE LEVEL: Please provide a rating oof your expertise in the area addressed by the paper, using the scale: 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average I'd say 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it extremely well Numeric Rating: 4 NB: PC members are responsible for ensuring that 1 is not used here. 5. ORIGINALITY. What is NEW and SIGNIFICANT in the work reported here? Comment: Presumably an improvement of a system called TUGEN which one of the authors have worked on before. 6. How WORTHWILE is the goal of the authors? Comment: It is probably an improvement of their previous algorithm but so what? 7. How well is this goal EXPLAINED and JUSTIFIED? Comment: Not very well 8. TECHNICAL QUALITY. Are the technical parts (definitions, statements, specifications, proofs, algorithms, etc.) SOUND? Comment: The pheudo-code looks all right 9. APPLICABILITY. If the work is primarily theoretical or conceptual, are the implications in terms of applicability adequately justified? If the paper is about a new formal technique, are satisfactory arguments presented in favor of the new technique? If a methodology is proposed, is it sound? If experimental results are presented, is their relevance justified? Comment: No 10. PRESENTATION: Describe the QUALITY of the writing, including suggestions for changes where appropriate. Comment: Poor 11. Were there any formatting or mechanical problems with this paper?: figures look strange Are the figures and length acceptable?: it is only 8 pages but it is more than enough Are the references correct?: probably 12. OTHER COMMENTS you believe would be useful to the author(s), including pointers to missing relevant work: I think that this paper has been submitted to the wrong conference. In addition it would be recommendable to get a native speaker reading it to get rid of the worst mistake in the English formulation. +++++++++++++++++++++ End of FME 2001 Paper Review Report ++++++++++++++++++++++ PAPER NUMBER: 05 CATEGORY: 2 TITLE: Towards execution in automatic test suite generation AUTHOR(S): Yixin Zhao Jianping Wu -------------------------------------------------------------------------------- 1. Briefly SUMMARIZE the paper (2-3 lines): The paper discusses an optimisation of TUGEN, an automatic test suite generation system. It replaces the old algorithm for "parameterizing & executing" in the TUGEN system. The partial experiments indicate that this produces better results. The basic idea is to use a formal specification which is given as an (extended kind of) transition system and then select certain transitions to be used as test cases. The parameters of the transitions (PDU fields, timer values etc.) are specified in a constraint-oriented way and as a consequence, the complexity of the constraint-solving procedure becomes a bottleneck in the process of test generation and execution. This problem and one solution to it are discussed. The paper is not well-written however, and the specification formalism is poorly explained (no formal semantics, for example). The examples are not self-contained. Claims are hardly verifiable. 2. RELEVANCE: Please provide a rating of the paper's relevance to the FME Symposium, using the scale: 0 = Out of scope 1 = Marginal interest 2 = Minority interest 3 = Majority interest 4 = Outstanding interest Numeric Rating: 3 Please comment your rating: Improving the efficiency of software testing is highly relevant (but the paper is weak). 3. OVERALL RECOMMENDATION: Please provide a rating of the paper's acceptability, using the scale: 1 = Strong reject 2 = Weak reject 3 = Could go either way 4 = Weak accept 5 = Strong accept Numeric Rating: 2 NB: There should be a correlation between the two rates above. Although the subject is interesting for the conference, the paper does not contain a sound theory because some explanations and proofs are missing. 4. CONFIDENCE LEVEL: Please provide a rating oof your expertise in the area addressed by the paper, using the scale: 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average I'd say 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it extremely well Numeric Rating: 4 NB: PC members are responsible for ensuring that 1 is not used here. 5. ORIGINALITY. What is NEW and SIGNIFICANT in the work reported here? Comment: It is an optimisation of an existing test suite generation system (TUEGEN). Experiments suggest that it works better than the old algorithm. 6. How WORTHWILE is the goal of the authors? Comment: because the replaced part of the TUGEN algorithm does not work very well, it seems worthy to try to improve it; so the goal of the authors is worthwhile. 7. How well is this goal EXPLAINED and JUSTIFIED? Comment: poor: the descriptions of the specification language and the algorithms are incomplete. 8. TECHNICAL QUALITY. Are the technical parts (definitions, statements, specifications, proofs, algorithms, etc.) SOUND? Comment: no, the theory does not seem sound. Some remarks: 1) Section 5.2 says: "A2 is complete in theory" I) in what sense? II) where is the proof? 2) In Section 2: Def. 1 uses elements which are defined in Def. 2 and Def. 3. We propose to switch the order. In Def. 3. "Z" is not introduced. It is a part of Rij. 3) In Section 3.1: "right_value=rightvalue -1" But "right_value" is a variable of any type. What means then CharString -1? 9. APPLICABILITY. If the work is primarily theoretical or conceptual, are the implications in terms of applicability adequately justified? If the paper is about a new formal technique, are satisfactory arguments presented in favor of the new technique? If a methodology is proposed, is it sound? If experimental results are presented, is their relevance justified? Comment: from the mathematical point of view, the work is not sound (at least, we cannot tell from the current paper). The experimental results indicate that the new algorithm works better than the old one. 10. PRESENTATION: Describe the QUALITY of the writing, including suggestions for changes where appropriate. Comment: about the English: there are many strange words such as satiability, automatilization, executizing etc. 11. Were there any formatting or mechanical problems with this paper?: Are the figures and length acceptable?: Are the references correct?: Figure 2: the process of A.1 : the decision blocks : the words "result, true, processed" intersect the lines of the boxes. For the rest, it seems O.K. 12. OTHER COMMENTS you believe would be useful to the author(s), including pointers to missing relevant work: Section 5.1 gives the description of the A.2 algorithms. Here we would expect a text which explains the algorithm (which is not the case, however). The definitions should be more explicit; an example is needed to illustrate the working of the protocols and the algorithms. The authors only refer to some experiments. +++++++++++++++++++++ End of FME 2001 Paper Review Report ++++++++++++++++++++++ PAPER NUMBER: 05 CATEGORY: 2 TITLE: Towards execution in automatic test suite generation AUTHOR(S): Yixin Zhao Jianping Wu -------------------------------------------------------------------------------- 1. Briefly SUMMARIZE the paper (2-3 lines): The paper covers the generation of test for protocols. A simple algorithm for test generation is investigated and problems with the algorithm identified. An improved algorithm is presented and shown to have overcome the problems. 2. RELEVANCE: Please provide a rating of the paper's relevance to the FME Symposium, using the scale: 0 = Out of scope 1 = Marginal interest 2 = Minority interest 3 = Majority interest 4 = Outstanding interest Numeric Rating: 3 Please comment your rating: I think that test case generation from formal specifications is a key technique to allow formal methods to be part of general software development. The work presented in the paper is very specialised but looks as if it could be generalised. 3. OVERALL RECOMMENDATION: Please provide a rating of the paper's acceptability, using the scale: 1 = Strong reject 2 = Weak reject 3 = Could go either way 4 = Weak accept 5 = Strong accept Numeric Rating: 4 NB: There should be a correlation between the two rates above. 4. CONFIDENCE LEVEL: Please provide a rating oof your expertise in the area addressed by the paper, using the scale: 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average I'd say 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it extremely well Numeric Rating: 3 NB: PC members are responsible for ensuring that 1 is not used here. 5. ORIGINALITY. What is NEW and SIGNIFICANT in the work reported here? Comment: The algorithm for the generation of a complete set of test cases for a protocol. 6. How WORTHWILE is the goal of the authors? Comment: The goal is very focused and worthwhile in its area. 7. How well is this goal EXPLAINED and JUSTIFIED? Comment: Reasonably 8. TECHNICAL QUALITY. Are the technical parts (definitions, statements, specifications, proofs, algorithms, etc.) SOUND? Comment: Generally ok, but example that A1 is not sufficient is not clear. 9. APPLICABILITY. If the work is primarily theoretical or conceptual, are the implications in terms of applicability adequately justified? If the paper is about a new formal technique, are satisfactory arguments presented in favor of the new technique? If a methodology is proposed, is it sound? If experimental results are presented, is their relevance justified? Comment: Work is directly applicable to protocols that can be expressed in External Behaviour Expression. May be applicable more generally. 10. PRESENTATION: Describe the QUALITY of the writing, including suggestions for changes where appropriate. Comment: Understandable but not fluent. 11. Were there any formatting or mechanical problems with this paper?: Some strange marks around special characters. Are the figures and length acceptable?: Length well within limits, figure could be larger and clearer. Are the references correct?: Yes 12. OTHER COMMENTS you believe would be useful to the author(s), including pointers to missing relevant work: None +++++++++++++++++++++ End of FME 2001 Paper Review Report ++++++++++++++++++++++ PAPER NUMBER: 05 CATEGORY: 2 TITLE: Towards execution in automatic test suite generation AUTHOR(S): Yixin Zhao Jianping Wu -------------------------------------------------------------------------------- 1. Briefly SUMMARIZE the paper (2-3 lines): Reports on an incremental improvement to an automatic test suite generation tool. It starts with a theoretical analysis of the tool and identifies improvement to the generation algorithm. Then, the theoretical improvement is realised through implementation. 2. RELEVANCE: Please provide a rating of the paper's relevance to the FME Symposium, using the scale: 0 = Out of scope 1 = Marginal interest 2 = Minority interest 3 = Majority interest 4 = Outstanding interest Numeric Rating: 2 Please comment your rating: The paper provides an interesting example of a formal approach to incremental development. The fact that the system being implemented is a test generation tool is not central to the article. 3. OVERALL RECOMMENDATION: Please provide a rating of the paper's acceptability, using the scale: 1 = Strong reject 2 = Weak reject 3 = Could go either way 4 = Weak accept 5 = Strong accept Numeric Rating: 2 Technically, the paper is not very strong, and the results are quite weak. 4. CONFIDENCE LEVEL: Please provide a rating oof your expertise in the area addressed by the paper, using the scale: 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average I'd say 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it extremely well Numeric Rating: 3.5 NB: PC members are responsible for ensuring that 1 is not used here. 5. ORIGINALITY. What is NEW and SIGNIFICANT in the work reported here? Comment: The refinement/optimization of their algorithm and the analysis of completeness. 6. How WORTHWILE is the goal of the authors? Comment: Always worthwhile to make such incremental development more rigorous/formal. However, this is a particular instance which is not generally applicable. 7. How well is this goal EXPLAINED and JUSTIFIED? Comment: OK 8. TECHNICAL QUALITY. Are the technical parts (definitions, statements, specifications, proofs, algorithms, etc.) SOUND? Comment: Some of the reasoning about complexity needs to be tightened up. In particular, the last paragraph of 4.2 needs to address the issue of 'wish' with more formality. It is not convincing as it stands. 9. APPLICABILITY. If the work is primarily theoretical or conceptual, are the implications in terms of applicability adequately justified? If the paper is about a new formal technique, are satisfactory arguments presented in favor of the new technique? If a methodology is proposed, is it sound? If experimental results are presented, is their relevance justified? Comment: Yes, but the results in optimization are not significant and the conclusions vague. The completeness argument is stronger although its importance can only be appreciated if the complexity aspects are more thoroughly explored. 10. PRESENTATION: Describe the QUALITY of the writing, including suggestions for changes where appropriate. Comment: VERY POOR ... Every other sentence appears to contain a spelling, punctuation or some other grammatical error. This led to difficulties in comprehension. 11. Were there any formatting or mechanical problems with this paper?: Are the figures and length acceptable?: Are the references correct?: Text in diagrams seemed to large for the graphics 12. OTHER COMMENTS you believe would be useful to the author(s), including pointers to missing relevant work: What is exactness in your final conclusions? A diagram (similar to fig. 2) for the 2nd algorithm would have helped in making a comparison. +++++++++++++++++++++ End of FME 2001 Paper Review Report ++++++++++++++++++++++