Benchmarking Methodology WG Minutes WG Chair: Kevin Dubray Minutes reported by Cynthia Martin. 1. Administrative Overview. Kevin Dubray provided an overview of the meeting agenda. No questions or comments were given from the audience. 2. Overview of ID "ATM ABR Benchmarking Terminology". Jeff Dunn provided an overview of the "ATM ABR Benchmarking Terminology" ID including updates and changes to the ID from previous versions. There is interest in the group to create an "ATM ABR Benchmarking Methodology" ID, but no proposed dates or timelines were discussed. Jeff requested that anyone interested in creating the methodology ID should contact Jeff at Jeffrey.Dunn@worldnet.att.net or via the BMWG mailing list. We need people with operational experience to provide input. The "ATM ABR Benchmarking Terminology" ID will go into last call after this BMWG meeting if no comments or questions are raised. 3. Overview of ID "Frame Relay Benchmarking Terminology". Jeff Dunn provided an overview of the "Frame Relay Benchmarking Terminology" ID including updates and changes to the ID from previous versions. The comment was made to add additional PDH (DS-3) and ADSL information in this ID or to create a new ID. This issue will be put on the list so everyone may comment on what they feel is the appropriate solution. The authors believe that a separate ID should be created for ADSL. The "Frame Relay Benchmarking Terminology" ID will go into last call after this BMWG meeting if no comments or questions are raised. 4. Overview of ID "ATM Benchmarking Methodology". Jeff Dunn provided an overview of the "ATM Benchmarking Methodology " ID including updates and changes to the ID from previous versions. Many comments and questions were raised including: a) Should the required histograms be normalized or absolute? After a discussion, it was decided that both should be available to the user and that a uniform format is more important. The ID will reflect this decision. b) How do we test cell rate margin? The Cell Rate Margin is defined the terminology ID, but we have not come up with a good test scenario for this. The authors would like to remove this test from the ID. c) The filter definitions are still not defined. Should we include them in the ID? Are the filters for ATM or IP? Does it effect throughput? These questions will be put on the list so everyone may comment. We need an answer to these for the final version of the ID. The authors would like to remove this from the ID. The "ATM Benchmarking Methodology" ID will go under one more revision before the next IETF and then into last call. 5. Overview of IP QOS ID proposal Jerry Perser provided an overview of the "IP QOS" ID. Many comments and questions were raised including: a) Many people are interested in the ID, but who is willing to work and comment on it? b) IP QOS is not defined or understood, how will this ID be developed? c) Should we address DIFFSERV? How does this interact with IP QOS? DIFF SERV does not define specific metrics or yardsticks for service agreements. How will this effect the development of the ID? d) Should we address RSVP? How does this interact with IP QOS? e) Should we address TOS? How does this interact with IP QOS? f) Should this be a configuration effort, referring to other work and re- interpreting existing metrics? g) Should we also address IP over ATM QOS and IP over Frame Relay QOS to narrow the scope? h) Is this tied to certain mechanisms and algorithms? i) Is this ingress traffic policing or something else on the egress? Is this traffic policing conformance? The attendees agreed that the goals of the proposal need to be reworked and articulated to the BMWG mailing list. 6. Overview of Routing Metrics proposals a) Guy Trotter provided an overview of the proposed "Forwarding Performance based on Forwarding Table Size" ID. A few questions were raised including: i. Will the ID address forwarding table or route table? The author clarified that the ID will address forwarding table only, the nomenclature needs to be consistent. ii. Will the ID address issues like how soon after the routing distribution is complete will traffic forwarding occur? The ID needs to address this. iii. Will the ID address the effects of how the routing entries are placed into the table and if it effects traffic forwarding? What are the boundaries on the ID? The ID needs to address this. b) Jeff Dunn provided an overview of the proposed "Framework for IPv4 Routing Benchmarking" ID and "IPv4 Routing Metrics" ID. A few questions were raised including: i. Will the ID address MPLS or QOS? The author clarified that the ID will not address QOS that is for the QOS ID discussed earlier. The ID will not address MPLS at this time. This will be put on the list so everyone may comment. The authors believe that MPLS should be discussed in another ID, similar to the ATM and Frame Relay ID's. ii. What is the scope of the ID? The ID needs to be very clear. With more people working in the ID, the authors believe that a clearly defined, well-scoped ID can be created. iii. Does this proposed ID overlap with the proposed "Forwarding Performance based on Forwarding Table Size" ID? Yes. The proposed "Forwarding Performance based on Forwarding Table Size" is dependent on the proposed "Framework for IPv4 Routing Benchmarking", but they can be developed in parallel. The framework is necessary to develop all other router based benchmarking ID's. c) Although a consensus was not taken, the group agreed that both proposals could be developed at the same time. Since the BMWG meeting, the authors of both ID's decided to work together on the "Framework for IPv4 Routing Benchmarking" ID. A first draft of this ID should be available in 1-2 months from the BMWG meeting date. A first draft of the proposed "Forwarding Performance based on Forwarding Table Size" ID was not discussed, but after the meeting the author did indicate that a first draft would be available within a few weeks from the BMWG meeting date.