For those who might be interested in today's 1.5 hour call on the GNSO review. It was lively. From the NCUC perspective, I think it is fair to say that we did not think the Westlake representative was taking our concerns as seriously as they should have, but we have managed to get more time to review the report again before it goes out for public comments. Stephanie Perrin -------- Forwarded Message -------- Subject: [gnso-review-dt] MP3 recording GNSO Review Working Party teleconference Tuesday, 03 March 2015 20:00UTC Date: Tue, 3 Mar 2015 23:51:57 +0000 From: Terri Agnew <[log in to unmask]> To: [log in to unmask] <[log in to unmask]> Dear All, Please find the MP3 recording of the GNSO Review Working Party teleconference held on GNSO Review Working Party teleconference Tuesday, 03 March 200UTC at http://audio.icann.org/gnso/gnso-review-03mar15-en.mp3 On page: http://gnso.icann.org/en/group-activities/calendar#mar The recordings and transcriptions of the calls are posted on the GNSO Master Calendar page: http://gnso.icann.org/calendar/ _Attendees:_ Jennifer Wolfe Ron Andruff David Maher Edward Morris Jennifer Standiford Klaus Stoll Amr Elsadr Robin Gross Rafik Dammak James Gannon Philip Sheppard Bill Drake Stephanie Perrin Mike Rodenbaugh Rudi Vansnick Tapani Tarvainen Sam Lanfranco Matthieu Camus Walid Al-Saqaf Ken Stubbs Laura Covington <https://community.icann.org/display/gnsocwgdtstwrdshp/Laura+Covington+SOI> _Guest speaker_: Richard Westlake _Apologies:_ Chuck Gomes Osvaldo Novoa _ICANN Staff:_ Mary Wong Marika Konings Larisa Gurnick Charla Shambley Lars Hoffman Glen de St Gery Terri Agnew ** Please let me know if your name has been left off the list ** Thank you. Kind regards, Terri Agnew *_Adobe Chat transcript:_* Terri Agnew:Dear all, Welcome to the GNSO Review Working Party on the 03 March 2015 Philip:Hello Jennifer Wolfe:Hello, Philip, and hello everyone, thanks for joining! Amr Elsadr - Observer:Hi all. Just to note that the NCSG members listed as members of the WP on the right-hand column of the screen is outdated. Rudi Vansnick:sorry for being late Terri Agnew:Welcome Rudi Mary Wong:@Amr, thanks for the note Amr Elsadr - Observer:@Mary: I'm not a member anymore. Stephanie and Bill now are. Mary Wong:Can we confirm which groups (if any) the observers on this call are? We note several participants who are not current Working Party members representing the GNSO SG/Cs. James Gannon (Silent Observer):James Gannon NCUC/NCSG Amr Elsadr - Observer/NCSG:Done. Sorry. Mary Wong:Thanks! Mary Wong:They have been sent to the list Terri Agnew:Welcome Ron Andruff Ron A:Apologies for my late arrival! Greetings all, Terri Agnew:Welcome Tapani Tarvainen Tapani Tarvainen:Thanks, and also aorry for being late. (NCUC/NCSG) Marika Konings:For those that would like to update their information (to include affiliation, or observer status), please note you can do so by going to the drop down menu in the attendee pod and select 'edit my info'. Robin Gross [observer-NCSG]:Unfortunately the faulty methodology used in the study makes much of the study not very useful. Terri Agnew:Welcome Sam Lanfranco Klaus Stoll:I do not seem to have received the comments from NCUC Terri Agnew:@Philip, your mic is not active Mary Wong:@Klaus, that may be an issue of server response/speed, so different WP members may receive messages at different times. It was sent to the mailing list by Glen. Klaus Stoll:Thanks Mary Terri Agnew:Welcome Matthieu Camus Matthieu Camus:Thank you... sorry to be late Larisa Gurnick:Stephanie Perrin, Phillip Sheppard and NCUC comments have been recieved in response to the working text so far. Matthieu Camus:Matthieu Camus, Internet Society France Larisa Gurnick:Please send your comments to [log in to unmask] <mailto:[log in to unmask]> Terri Agnew:Welcome Walid Al-Saqaf Richard Westlake:I'm sorry I can't reallly hear SP Richard Westlake:Bettee! Robin Gross [observer-NCSG]:As an active participant for more than 11 years, I would have liked to have been interviewed by Westlake (I did fill out the survey). Marika Konings:@Stephanie - from my experience, we have actually come a long way since 5-6 years ago where indeed many WGs had many of the same people involved. We've seen quite a few new people step up, including yourself, which I personally think is a really positive development. Amr Elsadr - Observer/NCSG:I still need to get around to joining one. :) Terri Agnew:Welcome Ken Stubbs Marika Konings:@Amr - last time I checked you were on quite a few ;-) Bill Drake:I should add I talked to people who started and gave up Amr Elsadr - Observer/NCSG::) Robin Gross [observer-NCSG]:who decided who got interviewed? Robin Gross [observer-NCSG]:where can we see the terms of reference? Larisa Gurnick:Please see https://community.icann.org/display/GR2/Community+Engagement+Meetings for a listing of extensive outreach and engagement activities Bill Drake:I and a number of other chairs were only interviewed after I complained about it on the SOAC chairs list Robin Gross [observer-NCSG]:I'd really like to know who decided who was interviewed? We need an answer to that question. Mary Wong:@Robin, the wiki page has the RFP: https://community.icann.org/x/aJLhAg Edward Morris (observer/NCUC/GNSO Council):We tried o get a good respomse, we failed, it is what it is, is not an acceptable response. Robin Gross [observer-NCSG]:where to begin with all the inaccuracies? Larisa Gurnick:@Robin - the working party provided guidance to Westlake throughout the process re: interview candidates Stephanie Perrin:It is extremely difficult to stream through the document quickly when juggling the scree, the chat, and our own comments. Mary Wong:To follow up on Larisa's point, staff provided Westlake at their request with lists of officers, past chairs, WG veterans etc. Larisa Gurnick:@Stephanie - I emailed a copy of the working text several minutes ago Mary Wong:Staff did not select or screen the interviewees, just to be clear. Bill Drake:thanks Mary. But chairs turned out not to be worthy Ron A:Sorry I dropped right after I finished my intervention. Dialing back in... Marika Konings:@Bill - I believe all chairs were invited early on, but not everyone caught the email ;-) Bill Drake:Marika, most chairs said they were not Bill Drake:we had a discussion of it on the SOAC list Klaus Stoll:How do we avoid that perceived inaccuracies are in themself inaccurate Bill Drake:actually I was told the staff discouraged interviews Robin Gross [observer-NCSG]:How did Westlake decide who to approach for interview? Bill Drake:Klaus, checking facts works Mary Wong:To clarify, by "chairs" the list included GNSO and WG chairs and vice-chairs as well as SG/C/SO/AC. Edward Morris (observer/NCUC/GNSO Council):Even if the invitations were such that's snowballing methodology and just is not acceptable for drawing larger inferences in a bounded community like the GNSO. Klaus Stoll:Bill, how about some facts not just "I was told" Bill Drake:I totally oppose the use of "I was told" in such a report Bill Drake:particularly when highly selective Robin Gross [observer-NCSG]:completely agree, Bill. Robin Gross [observer-NCSG]:The decision to negative stereotype NCUC is not appropriate. Marika Konings:@Mike - the Policy & Implementation WG is not reviewing the PDP, but has recommended a number of processes that may complement the existing PDP Mike Rodenbaugh:thx Marika, still there should be some mention in this doc? Marika Konings:I believe there are references to that effort in the text Robin Gross [observer-NCSG]:how were the interviews divided - equally among all constituencies? I wasn't interviewed. Marika Konings:for example page 30 Mike Rodenbaugh:thx. Marika. Re interviews. only one IPC member was interviewed. Seems like at least 10 from NCUC were interviewed.. Mike Rodenbaugh:So we share Robin's question Robin Gross [observer-NCSG]:sounds like questionable methodology, Mike, for making any recommendations. Bill Drake:I asked F2F in LA to be interviewed and was told no time Bill Drake:only after complaining on the SOAC chair list did I eventually get one Bill Drake:but this is a side issue Robin Gross [observer-NCSG]:the other issue is the choice to selectively handle comments about NCUC. Terri Agnew:Laura Covington has joined audio Robin Gross [observer-NCSG]:I'm (again) volunteering to be interviewed for this process. As a participant for more than 11 years, I can add something (ex-ncsg chair, ex-gnso-councilor, ex-ncuc chair). Edward Morris (observer/NCUC/GNSO Council):I'll volunteer to give Westlake questions on statistical sampling. Apparently Richard missedcthat class at Oxford. Edward Morris (observer/NCUC/GNSO Council):classes Larisa Gurnick:@Richard - please provide a list of people intervied. This will be included in the report - I understand. Mike Rodenbaugh:the list is already there in the Annexes Amr Elsadr - Observer/NCSG:These questions could all be avoided if a description of how informant selection could be included in the methodology description. Stephanie Perrin:THe list is there, who are they? What stakeholder groups? Mike Rodenbaugh:I agree that would be helpful Robin Gross [observer-NCSG]:Debra Hughes was interviewed? She hasn't been in NCSG for several years. Where did her name come from to be interviewed? Philip:Quotes reflect concerns and praise. They are the essence of good consultancy Marika Konings:@Stephanie - I believe that is information that still needs completing (note the column 'role' which I presume will identify their affiliation) Larisa Gurnick:@Richard - can you please clarify what information you plan to include in the "Role" column? Stephanie Perrin:And when did they arrive at ICANN? This is important, if you are looking at attracting newcomers, and you are not actually talking to any Marika Konings:@Robin - she was one of the founders of NPOC which is one of the GNSO constituencies Philip:Section 8i about constituencies. They only exist in the CSG and NCSG Robin Gross [observer-NCSG]:She left NCSG years ago. Where was the suggestion from to interview her (staff?) Marika Konings:@Robin - I believe there are other names that are no longer actively involved, but may provided some historic perspectives that were of interest to Westlake? Mary Wong:And to clarify/repeat - staff did not tell Westlake who to inteview and who not to. Robin Gross [observer-NCSG]:we should know WHO made the specific suggestions. rafik:@Marika better to explain the criteria that led to interviewes selection. for example I see many board members inclucded, looking to understand thw rationale Robin Gross [observer-NCSG]:so Westlake selectively decided which comments to publish, thanks for the clarification, Richard. Robin Gross [observer-NCSG]:I also want to know if negative comments were made about other SG/Constituencies or only about NCUC in Westlake's effort. Marika Konings:@Rafik - that is a question for Richard/Westlake to answer, but as I understand it, it was a combination of suggestions that may have been received from staff, community members as well as aiming to get further input from those that responded to the 360 assessment. Robin Gross [observer-NCSG]:why are so many staff included as interviewees? that is not appropriate. I'd like to see the results with those comments deleted. rafik:@Marika and that is why I am looking for westlake answer :) Marika Konings:@Rafik - I only saw Ray as a board member on the list ? Stephanie Perrin:Bill Graham is a recent board member rafik:@marika including former too Robin Gross [observer-NCSG]:Why didn't Westlake just go look at NCUC's webpage to see the diversity of its EC, etc? Why wasn't a pursuit for facts used (only perceptions)? Marika Konings:@Rafik - some of the former board members have served in other roles too, so their perspectives may have been of interest in relation to those other roles and not necessarily their former board position ;-) Mary Wong:Again, to follow up on Marika's point, the Board members listed as being interviewed are either past or present members of the SIC - so some of them would have been part of the prior restructuring exericise. Robin Gross [observer-NCSG]:The facts are readily available on the web. Robin Gross [observer-NCSG]:why not use facts. Robin Gross [observer-NCSG]:sounds like Westlake is relying on an entirely "subjective" determination. Why not also include objective facts? Edward Morris (observer/NCUC/GNSO Council):Will Westlake please comment on the statistical validity of a survey of 27 people commenting on a group of 404 people? Ron A:As I understand it, any person in the community had the opportunity to bring their views forward regarding any part of ICANN GNSO... James Gannon (Silent Observer NCUC/NCSG):It seems on a blind read of the situation that there is major concern over the qualitiative and subjective bias of the review compred to a fact based quantative review.. Robin Gross [observer-NCSG]:It is a Fact that 27 different people have been on NCUC's EC for the last 10 years. It is a verifiable fact that the NCUC EC includes 1 rep from each geographic region. Not using these facts seems questionable. Edward Morris (observer/NCUC/GNSO Council):I'll comment. The confidence interval of such a sample is 18.23% making the data completely useless. Philip:Comparing 27 to 404 in this context is completely useless. Edward Morris (observer/NCUC/GNSO Council):Thanks Philip. Robin Gross [observer-NCSG]:why did westlake decide to ONLY go the subjective route? Larisa Gurnick:@Robin - Westlake methodology includes multiple data collection methods - quantitative and qualitative. Sam Lanfranco [NCSG-NPOC]:@Morris Technically, the C.I . is 18.23% if there is confidence in a sample being representative. Robin Gross [observer-NCSG]:Larisa, Richard just told us they decided to go the subjective route. Otherwise, what objective facts were relied upon? Edward Morris (observer/NCUC/GNSO Council):@Sam. Good point. That number is the best case scenerio for the survey! Philip:A consultant's role is to report comments received. lets not dismiss this please Bill Drake:actually Philip I don't agree, the role is to do research Philip:Well maybe both Edward Morris (observer/NCUC/GNSO Council):@Philip. Comments received - I agree. I'm having a problem with how they take faulty quant methodology, use that to select "representative" comments...I guess I'm questioning the 'representative' aspect. That's something we really don'y know due to some very poor esearch design and methodology. Stephanie Perrin:There is certainly an obligation to rely on facts and filter for bias. No evidence of how potential bias is filtered, in a fractious multi-stakeholder community where it is required. Robin Gross [observer-NCSG]:good point, Stephanie. How was "bias" accounted for by Westlake? Charla Shambley:You can view the 360 Assessment here: https://www.icann.org/en/system/files/files/360-assessment-21aug14-en.pdf Stephanie Perrin:Thanks Charla, I have been through all that data. I can bring it to my university library and demonstrate the bias, but frankly I do not see that as my job. Robin Gross [observer-NCSG]:what criteria does westlake use for its "professional selection" of what to provide? Philip:Between bias and truth is perception. Bill Drake:I understood the criteria to be what they think is relevant Robin Gross [observer-NCSG]:and what do they think is relevant? James Gannon (Silent Observer NCUC/NCSG):@robin those criteria will be extremely important for everyone to understand any subjective professional selection reccomendations by westlake Ron A:Do I understand this right: Westlake is expecting comments on this draft - or a new one coming out? Robin Gross:On what evidence did Westlake base its recommendation that additional constituencies will increase participation? Bill Drake:faith based Stephanie Perrin:Totally agree with Mike Robin Gross:who is driving this bus any way? Bill Drake:totally agree Mike James Gannon (Silent Observer NCUC/NCSG):+1 Mike Ron A:Agree - 20-30 days for community review and response. Robin Gross:The community should be making this determination. Not staff. Philip:Agree it is more important t get this right than to get this done. Robin Gross:Who is in charge? The community? Staff? Westlake? Philip:Us surely Bill Drake:surely Robin Gross:Then we set the dates. Robin Gross:And we decide when our report is ready for prime tme. James Gannon (Observer NCUC/NCSG):Surely the WG need to be given time to review before the public comment period of the full and complete report Philip:Agree Bill Drake:I'd like to comment on a complete draft before it goes out the door for public comment Robin Gross:Frankly, we are going to need another entirely new report given the issues with this one. Bill Drake:"Quality trumps timelines" is a good mantra Robin Gross:And not exactly "bottom up" Edward Morris (observer/NCUC/GNSO Council):Agreed Robin. The survey is methodologically troubled it can not be fixed. Bill Drake:Yes Stephanie it would be good if it were understood why people volunteer their time Philip:Agree completely with SP Bill Drake:20 hours per week of pro bono to get abused is not really a good deal Robin Gross:Westlake works for us. We need to remember that. Ron A:@ Bill: you are referring to us, right?! ;o) Philip:In all ICANN hisory, this report on ICANN's policy development body is the most important. Bill Drake:Ron you got skin in the game! Laura Covington:agree with Stephanie and Mike R Edward Morris (observer/NCUC/GNSO Council):@Bill. Only 20? Robin Gross:Right, Phillip, this is enormously important. Why on earth rush it when the community is uncomfortable with methodology? Bill Drake:Civil society people are crazy to try to work here. We are treated better in the UN. Bill Drake:I mean that Robin Gross:Let's fix this and get it right. Bill Drake:Human Rights Council meeting now in Geneva. CS people really having productive engagement, and not being crapped on for their efforts. Stephanie Perrin:yes Philip:Bye all. A very productive meeting. New call with new drfat please. Robin Gross:The community has spoken on this, with a rare one voice. Terri Agnew:@Klaus, I see your mic is muted Robin Gross:I hope these errors can be fixed in the next draft. Stephanie Perrin:Thanks for chairing a difficult meeting!! Ron A:Thanks all! Bill Drake:thank you Jen, good chairing :-) James Gannon (Observer NCUC/NCSG):Thanks all. Rudi Vansnick:thanks Robin Gross:Thanks, all. Bye! Amr Elsadr - Observer/NCSG:Thanks all. Bye. *__*