++++++++++++++++++ FME2001: - Review reports for paper 36 +++++++++++++++++++++ - Dear author(s) : this file was extracted automatically from a very large - mailbox. In case you find any problem concerning mail encodings (or any - other kind of anomaly disturbing your understanding of the reviews) please - email any of the PC cochairs (pamela@research.att.com,jno@di.uminho.pt). ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ PAPER NUMBER: 36 CATEGORY: 1 TITLE: A Method for Systematic Requirements Elicitation: Application to a Light Control System AUTHOR(S): Jeanine Souquieres Maritta Heisel -------------------------------------------------------------------------------- 1. Briefly SUMMARIZE the paper (2-3 lines): The paper describes the application of the authors' method for requirements elicitation and formalisation (described elsewhere) in a case study. The authors are not the source of the case. 2. RELEVANCE: Please provide a rating of the paper's relevance to the FME Symposium, using the scale: 0 = Out of scope 1 = Marginal interest 2 = Minority interest 3 = Majority interest 4 = Outstanding interest Numeric Rating: 2 Please comment your rating: The topic as such has a higher relevance, this score is based on the limited applicability of the proposed formalisation. 3. OVERALL RECOMMENDATION: Please provide a rating of the paper's acceptability, using the scale: 1 = Strong reject 2 = Weak reject 3 = Could go either way 4 = Weak accept 5 = Strong accept Numeric Rating: 2 NB: There should be a correlation between the two rates above. 4. CONFIDENCE LEVEL: Please provide a rating of your expertise in the area addressed by the paper, using the scale: 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average I'd say 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it extremely well Numeric Rating: 4 NB: PC members are responsible for ensuring that 1 is not used here. 5. ORIGINALITY. What is NEW and SIGNIFICANT in the work reported here? Comment: Not very much. 6. How WORTHWILE is the goal of the authors? Comment: See 7. 7. How well is this goal EXPLAINED and JUSTIFIED? Comment: The goal of the paper can be formulated at several levels. a) a systematic approach to requirements elicitation that helps to detect inconsistencies, incompleteness, and ambiguities. (quote from the paper) b) to show that formal methods contribute to a) c) to show that the authors' method achieves a). These goals are well-explained, worthwhile and justified. The problem is that the paper doesn't meet in particular goal c). 8. TECHNICAL QUALITY. Are the technical parts (definitions, statements, specifications, proofs, algorithms, etc.) SOUND? Comment: Yes. However, see 9. 9. APPLICABILITY. If the work is primarily theoretical or conceptual, are the implications in terms of applicability adequately justified? If the paper is about a new formal technique, are satisfactory arguments presented in favor of the new technique? If a methodology is proposed, is it sound? If experimental results are presented, is their relevance justified? Comment: The authors fail to show why their method would achieve the goals better (in any sense) than any other formal method. To me it is quite obvious that any reasonable attempt to formalise the requirements reveals their shortcomings. Worse, the authors fail to lift the discussion from the case study to a more general discussion about the (de)merits of their method. See 12 for details. 10.PRESENTATION: Describe the QUALITY of the writing, including suggestions for changes where appropriate. Comment: By discussing one requirement at a time, some *issues* arising in 4.2 are scattered over multiple places. 11. Were there any formatting or mechanical problems with this paper?: no Are the figures and length acceptable?: yes, but a little lengthy Are the references correct?: yes 12. OTHER COMMENTS you believe would be useful to the author(s), including pointers to missing relevant work: There are several standard problems that a formal method for this kind of systems must tackle. I am concerned, and in some cases convinced, that your formalism does not tackle these problems. The article doesn't even mention them. (There are many references to such problems - perhaps Sandewall's "Features and Fluents" is a useful starting point.) The ramification problem. Suppose there are two lights A and B. One light is sufficient to ensure safe_illumination. - Does the event "turn off A" establish "not safe_illumination"? - If B is off, is the effect of "turn off A" that "safe_illumination" becomes false, or (presumably through intervention of the control system) that B is turned on? Indirect effects. Consider U1. The expression "occupied ~>safe_illumination" expresses the requirement. "enter_first ~> occupied" is a semantic relation in the environment. But why is the expression "enter_first ~>safe_illumination" there? That is definitely not a semantic relation: it is a derived requirement or a proof obligation. Too many warnings for possible interactions. When formalising U10, you "get as interaction candidates all requirements formalized so far". This is not acceptable behaviour: if you had an otherwise unrelated specification of the ventilation system, U10 would interact with those requirements as well. In fact, your Precondition Interaction Analysis need to be refined. The specification disallows environmental events. You follow correctly the separation of events in JZ95. However, in your definition of "immediately_followed_by", you specify that the next event is the system's reaction (no matter how long it takes in real time). There is also a problem of compositionality: if you take two independent copies of the system, an allowed trace of each copy, and combine these into a single trace (using the time stamp order), the combined trace may not be allowed. Modularity. Not so obvious in this small example, but in a larger system your specification formalism lacks modularity. Consider a distributed system (eg a telephone system). Suppose there are 10 events in component A that result in sending signal S to component B. Suppose that B can react to S in 10 different ways, depending on its state. You don't want to specify all 100 combinations separately. Thus you want to model signal S as an internal event (ie not shared by the environment). To conclude, I think that the (presentation of the) case in 4.1 is so obviously flawed that one can hardly deduce from it that your method (or any formal method) is better at finding ambiguities and incompleteness than other methods. Obviously the presentation is incomplete: sensors are supposed to generate other events than just their failure, and somehow the effect of pressing a button should have some effect on the lights (besides, isn't U9(i) related to this?). +++++++++++++++++++++ End of FME 2001 Paper Review Report ++++++++++++++++++++++ PAPER NUMBER: 36 CATEGORY: 1 TITLE: A Method for Systematic Requirements Elicitation: Application to a Light Control System AUTHOR(S): Jeanine Souquières Maritta Heisel -------------------------------------------------------------------------------- 1. Briefly SUMMARIZE the paper (2-3 lines): The paper demonstrates the use of a systematic approach to clarify and analyze the requirements of a system. The method includes the formalization of the requirements and the analysis of interactions between them. 2. RELEVANCE: Please provide a rating of the paper's relevance to the FME Symposium, using the scale: 0 = Out of scope 1 = Marginal interest 2 = Minority interest 3 = Majority interest 4 = Outstanding interest Numeric Rating: 3 Please comment your rating: A method like that is quite important in order to reduce the ambiguite in the understanding of users' needs. 3. OVERALL RECOMMENDATION: Please provide a rating of the paper's acceptability, using the scale: 1 = Strong reject 2 = Weak reject 3 = Could go either way 4 = Weak accept 5 = Strong accept Numeric Rating: 3 NB: There should be a correlation between the two rates above. 4. CONFIDENCE LEVEL: Please provide a rating oof your expertise in the area addressed by the paper, using the scale: 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average I'd say 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it extremely well Numeric Rating: 4 NB: PC members are responsible for ensuring that 1 is not used here. 5. ORIGINALITY. What is NEW and SIGNIFICANT in the work reported here? Comment: The authors have already published, in 1998 and 1999, three other papers about the subject and is not clear in the current paper what are the main differences among all the papers. 6. How WORTHWILE is the goal of the authors? Comment: Creating a method for formalizing and document requirements is quite important to reduce the gap between the users' needs and the requirement specification. 7. How well is this goal EXPLAINED and JUSTIFIED? Comment: I felt a bit of difficulty to understand all the concepts. The text is not so easy to read. 8. TECHNICAL QUALITY. Are the technical parts (definitions, statements, specifications, proofs, algorithms, etc.) SOUND? Comment: In spite of using formal methods, the specifications were not proved/ 9. APPLICABILITY. If the work is primarily theoretical or conceptual, are the implications in terms of applicability adequately justified? If the paper is about a new formal technique, are satisfactory arguments presented in favor of the new technique? If a methodology is proposed, is it sound? If experimental results are presented, is their relevance justified? Comment: The method is sound. However, it seems a bit heavy to be applied in practice. 10. PRESENTATION: Describe the QUALITY of the writing, including suggestions for changes where appropriate. Comment: The authors should have used more pictures to illustrate the case study. 11. Were there any formatting or mechanical problems with this paper?: No Are the figures and length acceptable?: Yes Are the references correct?: Yes 12. OTHER COMMENTS you believe would be useful to the author(s), including pointers to missing relevant work: It would be good if the correspondence between the method and other OO methods was explained more clearly. +++++++++++++++++++++ End of FME 2001 Paper Review Report ++++++++++++++++++++++ PAPER NUMBER: 36 CATEGORY: 1 TITLE: A method for systematic requirements elicitation: Application to a light control system AUTHOR(S): Souquieres, Heisel -------------------------------------------------------------------------------- 1. Briefly SUMMARIZE the paper (2-3 lines): Provides a method for formalizing informal requirements and detecting potential interactions among them. 2. RELEVANCE: Please provide a rating of the paper's relevance to the FME Symposium, using the scale: 0 = Out of scope 1 = Marginal interest 2 = Minority interest 3 = Majority interest 4 = Outstanding interest Numeric Rating: 3 Please comment your rating: 3. OVERALL RECOMMENDATION: Please provide a rating of the paper's acceptability, using the scale: 1 = Strong reject 2 = Weak reject 3 = Could go either way 4 = Weak accept 5 = Strong accept Numeric Rating: 4 NB: There should be a correlation between the two rates above. 4. CONFIDENCE LEVEL: Please provide a rating oof your expertise in the area addressed by the paper, using the scale: 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average I'd say 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it extremely well Numeric Rating: 5 NB: PC members are responsible for ensuring that 1 is not used here. 5. ORIGINALITY. What is NEW and SIGNIFICANT in the work reported here? Comment: The approach to detecting potential interactions among requirements is new and interesting. If it works well, it would be very worthwhile. 6. How WORTHWILE is the goal of the authors? Comment: 7. How well is this goal EXPLAINED and JUSTIFIED? Comment: 8. TECHNICAL QUALITY. Are the technical parts (definitions, statements, specifications, proofs, algorithms, etc.) SOUND? Comment: The big danger of an approach like this is that, because of the abstraction used to achieve practical interaction analysis, the results will be too coarse. Everything will have potential interactions with everything else, and thus the analysis will prove useless. I don't know if the method presented here suffers from this problem or not. The presentation is not good enough for me to judge. As a result, I like the paper but am uneasy about accepting it. 9. APPLICABILITY. If the work is primarily theoretical or conceptual, are the implications in terms of applicability adequately justified? If the paper is about a new formal technique, are satisfactory arguments presented in favor of the new technique? If a methodology is proposed, is it sound? If experimental results are presented, is their relevance justified? Comment: 10. PRESENTATION: Describe the QUALITY of the writing, including suggestions for changes where appropriate. Comment: It is very difficult to tell how this works. The problem is that the method is presented in Section 2 with a lot of formalism and no example. The explanation is full of abstract, ambiguous words such as "independent." Then, in Section 4, the example is presented very briefly and informally. There is little direct correlation with the formalism. For example, the authors say which requirements are interaction candidates, without going through the computation of them as prescribed in Section 2. The critical thing that I need to understand what is going on is a good intuitive example that follows the formalism step-by-step. This is completely missing. Other questions: 1) Are there any frame conditions in your formalism? It seems so, but they are not mentioned. 2) I am very puzzled as to how the event "leave_last" can be shared with the software system. How is it detected? 3) In the middle of page 13 we find the semantic relation turn_off ---> not same_light_scene Wouldn't this be wrong if the same_light_scene is dark? Isn't this a case where all semantic relations are possible and thus the semantic relations give no information? 11. Were there any formatting or mechanical problems with this paper?: Are the figures and length acceptable?: Are the references correct?: 12. OTHER COMMENTS you believe would be useful to the author(s), including pointers to missing relevant work: +++++++++++++++++++++ End of FME 2001 Paper Review Report ++++++++++++++++++++++