Earlier this week, the PASS speaker portal was updated with the session evaluations from this year's PASS Summit in Charlotte, NC. Most people were very happy with their evaluations, though a few were pretty hard on themselves. One thing I wanted to say was:
Stop worrying about your ranking!
It's hard to avoid comparing your score to the rest of the field. But I have to point out that almost the entire conference was above 4.0 on a 5-point scale (and more than half was above 4.5). That is truly remarkable, and is a testament to the quality of both the speakers and the job done by the selection committee in vetting speakers and sessions. This really was a high-quality event and, depending on the size of your audience, your rank amongst such an esteemed group of peers could be crushed by a single grumpy reviewer giving you a 3 or 4 instead of a 5. Maybe you made fun of Canadians and they didn't like it, or maybe they blame you for the temperature of the room or that your talk was in a small room and they had to stand. So don't take your ranking too seriously: focus on the qualitative feedback about your content and delivery, and try to remember the things you know you need to improve on. Work on those things for next time; don't work on how to jockey a few places in the rankings so you can be ahead of so-and-so.
Now, since I know that people will still want to dwell on the numbers, there was a little snafu in this year's survey which threw off the original overall scores and rankings. Question #6 asked:
Did you learn what you expected from this session?
And then the possible answers were:
1. Yes 2. No 3. Sort of
This is not a sliding scale: 1 is best, 2 is worst, and 3 is mediocre. Kendra Little (@Kendra_Little) put it best in this graphic.
So it's ok to collect information in this way if you're just going to count up the Yesses, Nos and Sort ofs, but it can't be used to calculate a numerical score. For example, the average for one of my sessions was 1.18. What does that mean? Did I have five people who learned nothing from my session, or two people who "sort of" learned what they expected? Calculating an average here, even on its own, was entirely meaningless; worse yet, those with better scores here actually pulled their overall score *down* further when it was included in the overall score calculation. To their credit, PASS was quick to correct this (unlike the other survey snafu); within about two hours of publishing the results, they had recalculated the scores, dropping Question #6 altogether:
Now, my suggestion is, if you really want to know your overall score while incorporating the data from Question #6, just change the scale. Make Yes a 5, No a 1, and Sort of a 3. You can do this by downloading the CSV, bulk inserting it, and updating the data directly. Or if you're a really good Excel wrangler, you could probably just do it there. I tried to automate this but I really suck at Excel and the CSV format is not really made for bulk insert (some of the columns are actually separate by CR/LF, not commas). So you may need to do some work to get it into a format you can work with, but the idea is there, anyway.
Aaron,
Thank you for writing this. I also don't get it either. What happened to the good old days where we focused on connecting, sharing and learning?
I will take it one step further. I actually don't care about the numbers at all. I only care about the comments. How are you honestly going to judge the numbers? My 5 might be your 3 and we might actually think the same thing about the session.
Regards,
John