Should there be a change in how Judges score on the Weekly conte

This is a continuation of the discussion that was initiated in the weekly winners thread of 12-19-11.

[IMPORTANT: To the new reader: The main suggestion in this post has been resolved..see my 7th post .....Though the idea sounded good to begin with, it is not mathematically feasible/correct...so is ABANDONED. The thread also has other ideas in it that are still brewing....but may or may not have any resolution to it...so be WARNED on wasting your time!]

This issue arose when PnQ members felt that there are simply too many ties in the weekly contest for the 3rd place, because of the current scoring system (5 judges, who score only on the 3 top poems of their choice using 10, 7 & 4).

So here are some thoughts of how the judging/scoring system may be modernized, given its current pitfalls. My original thoughts, are copied from the post above. I would suggest that moderators take the lead and determine whether changes are needed, would they solve the issues, and whether they can be implemented easily, if there is a consensus.

FIRST POST:

All the poems on the nomination should be voted on, not just the 3 best one's. The scoring could be on a 1-5 scale as follows:

1= subpar
2= average
3= Above average
4= Superior/Very Good
5= Outstanding/Excellent

When judges vote on all the poems it allows for a much closer competetion, a lot more fairness when total scores are calculated, and a dilution of favoritism or pay back wins (it does exist in our psyche, regardless of how much we pretend to be fair).

Here are addl. suggestions to the above:

1) Judges should only be allowed to use 5 x 2, 4 x 2, (or any similar scheme by consensus) but can use 3,2,1 unlimited times, to score all the nominated poems. The limitation on use of 5's and 4's will disallow the judges picking a 5 for all their winners (was a concern of one of the mods).
This way an individual judge will not be pressured to pick just 1 top and 1 second poem, as with more and more good quality poems, it is probably increasingly difficult for them to pick just 1 top poem.

2) As all the other remaining 4 judges will also be forced to vote on your top rated poems (the two 5's and the two 4's), ties would be very unlikely.....though by laws of probability, every once in a while (not every week) it may happen. At which point, one can use other criteria such as counting the # of votes/comments to break the tie.

I am curious what other senior (and junior)members who have been here a long time think about this suggestion, or if this has been already hashed and trashed in the past, then let it go out again.

SECOND POST:

Addl advantages (did not want to publish them initially...but given the nature of the qns raised):

1) The mandatory scoring on all poems can easily identify downvoting (people not in our good books)and upvoting (club members, "populars" and those on our favorite list, for others to see (e.g. mods)....which will then allow them (biased judges) to be hopefully unseated (no offense implied or to go on the defensive....it is just a natural human tendency.....ask any psychologist, how they identify it when they administer certain tests). This process can remain confidential, so no one's pride is hurt in the public.
Secondly, it will also dissuade downvoting, as judges will likely not downvote a poem that is otherwise outstanding to many.......this is your next best way, as it is impossible to have all poems to be anonymous to the judges.

2) Eliminate the judge's dilemma, when he/she sees 2, or 3 or even 4 very good poems, and has to rank them as 10, 7 & 4 on the current system. (Reiterated from point 1 above).

3) And to incorporate Sibyllene's comment (from weekly winners thread) a single 10 is still > 4+4, (i.e a single judge's winner can outshine 2 judges 3rd winner!)

3) Comments still need only be written on the top 2, 3 or 4...that can be decided by a consensus. So no significant addl work.

4) Agree with Abby (from weekly thread) on this....to have Janis add a total score display for all poems, so the authors get a feedback on their writings. DOWNSIDE of letting that happen before the contest is over, is biasing the judges voting last, by telling them how their colleagues are voting...and then we all know how human psychology, defines "pursuasion".

5) Re: Sibs comment "managing all the scores will probably be a huge bugger for whoever is running the contest".
Probably not any more work, as the computer totals, will pick the top 3 and as ties are going to be less probable, the person verifying the scores will just have to identify and verify the tally from what is mailed to them, by the judges. However, if there are 3,2,or 1 contributing to the winner's scores, you have to trust the programmer's adding system. There will not be any comments associated with those receiving 3,2 or 1....so there will be no need to worry about identifying the comments from these judges to be posted in the weekly blog on winners.

6) HM's will not change, as all comments from the judges, for the non-winner's are still available to be posted in the weekly thread. But ties will be practically eliminated.

Lastly, if one think's of this new concept logically, it is not at all complicated, to implement and follow. Needs very little in the way of programming difficulties.

I've never been any good at maths so please forgive this question...I believe we have 5 judges right now...so say two of them give one poem 5 each...5+5 = 10..then then all five judges can randomly give out 2's so...the 2 judges who gave out 5's still have the opportunity (along with the other three to vote 2's on another poem)..5x2 = 10...tie break situation again?

I'm not really for giving out 1,2 and 3 votes...but I think for the judges...having an opportunity to give 5 twice is basically a good idea.

One of my concerns was that people might just vote a "5" on all of their buddies, if they had unlimited 5 votes to give away"

^^

quote from a mod..

Is that a show of confidence?

Edit

Anyway...I'm going to be busy this next week so...Merry Christmas to you all..a little ahead of of time I know :)

My point with number 5 would be that Nana (or whoever) would somehow have to make sure that each judge was voting on each poem. It would be easier to accidentally skip one when there are so many.

I still don't really see how it would discourage up-voting and down-voting, but the rest seems pretty solid to me.

"Is that a show of confidence?"

Nope, not really. Call me a cynic but... I think judges all have honest intentions but could be unconsciously tempted to use "extra" votes to boost their friends or favorites. However, like I said, with a cap on "5" votes I don't see that as being a big issue anymore.

These suggestions sound interesting and would probably help decrease the number of ties. They have their advantages but while I think a change would be good, I can see some small probelms with the 'out of 5' method of voting.

First, I dont think it would change anything with regard favouritism voting: not saying this happens but if a judge wants to vote for a friend etc this system wont really stop them, even if they only have two top votes, they will be able to give them to their friends if they so want to. A restiriction would help but it wouldnt really change or eradicate any intentional or unintentional bias.

Also, there are more than 5 degrees of quality of a poem in my opinion. Say for example a judge thinks one poem stands out and gives it five. Wouldnt it be unfair on that poet and poem if the second best poem got voted the same amount? So say then the judge votes the second best poem 4pts. Same problem, if the third best poem gets voted 4pts, isnt it unfair on the second best poem to have an inferior poem in the eyes of the judge be voted the same amount of points?

I would suggest at least changing the points distributed to 5pts, 4pts and 3pts as opposed to 10, 7, 4. When I was a judge, I found there was a huge gap between 4pts and 10pts between poems that were usually close in quality. With the 5pts, 4pts and 3pts for example, two votes of 3pts would beat one poem of 5pts, which is probably fair.

An extension of this idea would be to give the judges the power to vote 1,2,3,4 and 5points each week. They would have to use three of their votes and could use all five, depending on week to week. Say one week there were not many good poems nominated, a judge might vote 5pts 3pts 2pts on the three poems. If in another week the quality was close or there were many good poems, the judge could use all five votes, 5,4,3,2,1. It would give the judges power to reward poems they see as exceptional, or reward poems that were very close to making the top three by giving their extra votes to. But I think the major fault in this plan could out-weigh the advantages: one judge having more of an influence than another. Also it wouldnt help bias voting if it existed.

An definite improvement to the contest would be to hide the amount of votes each poem has so that the judges dont know how many votes each poem already has.

An anonymous system would still be the best if it were possible!

From a conversation on a previous thread continued....

LP...this is your quote from a past win...and I might
add this poem won where one judge gave you a ten without comment..

"Now I can say, I HAVE WOOONNN :p lol
I must leave this talent behind. I've just got what I yearned for".

You seemed quite happy that week..yearning for something then... now telling others not to cry about unfair judging??? I'm not yearning...nor am I crying...just looking at the system we have and thinking...bollocks!

The week after that BTW...another of your poems was in a 4 way tie for 3rd place and...you won again...I do know the judges changed in that week but...it was one of the first tie situations I've come across...now they are more common...wonder why...also I've noticed that many poems that get an HM one week go on to win the next week...I find that strange also? Maybe this all needs to go to the other thread opened by A Lonely Soul? Yeah..I'll leave this comment in answer to you and I'll also post it in his new one!

Hellon, i dont see where ur going.

What u said is so irrelevant, what does being happy about a win have to do with the system ur annoyed about?

It just another unfair system. Deal with it.. Deal with it.

LP some like you put more and more credence into getting a win beside one of your poems than most, in fact you went as far as to post a poem with an imaginary [win] against it. It seemed like your only focus for being here, so what Hellon is saying is very relevant. Your condescending response is flippant immature and uncalled for. And her name is Hellon, or should we all resort to name calling.

In regards to this thread, I've never worried too much about what's on the front page but A Lonely Soul has a valid point

^ cool down man!

I think you don't have a sense of humor, for you never understood my poem.

The Poem of All Poems [WIN]

^ It was a mockery poem for all people out there who thinks that having a WIN beside their poem can make a difference. If you have read well, the end, you would have understood ;) but hey... whatever.

And, my God, I swear I was on the phone and now I noticed that I wrote Hellong.

at any level, Hellon knows how much I love her, and her writings... There's no need for you to defend her ;)

Hey, let's not fight on Christmas. Santa might still be watching....

Hellong, in response to a couple of your comments:

"it was one of the first tie situations I've come across...now they are more common...wonder why..."

I'm finding with this batch of judges that they each have very distinct preferences, and so have been voting for a wide variety of poems. I think this is a good thing, but a result is that there have been less "doubling up" on votes. Judge A will pick three completely different poems than Judge B. So we're getting a wide spread. We want judges to vote for their favorite poems, of course, so we wouldn't want to change what they should vote for, but it's because of situations like this where it becomes evident that we may need a new judging system. Normally the judges are more in accordance with each other, so it's not as much of an issue.

"also I've noticed that many poems that get an HM one week go on to win the next week...I find that strange also?"

I think this is normal. If someone liked a poem enough to vote for it one week, they may be likely to vote for it again. And a judge who liked other poems more the previous week may find an HM that is now better than the other poems in the pile, once older ones are cleared out or have won.

^ Now that's what I'd love to hear. I agree with Sib.. but hey, Sib, don't make from my mistake, a sin.

LP we can see through you like a window pane.. whats not cool is you man lol

Hi Sibs happy boxing day... I'm not fighting just making an observation that we all can see

Regarding the voting system I'm not to concerned either way...

LP we can see through you like a window pane.. What's not cool is you man lol

^ thank you, and your X-Mas spirit.

merry X-Mas all

Sure the judges now have a specific preference for very different poems.. but it doesn't mean the next batch of judges will. What if we change it and the next set of judges make a zillion ties? We cant change it each time.

I do think the system needs revamping, but I don't have specific suggestions. My only thing would make.the points closer together.

Nothing wrong with my X-Mas spirit dude...im all happy and smiley...

What if a poem was only allowed two nominations each? The whole idea of these nominations is to get a poem noticed and, if it's by a member who is unknown, and may not be read by so many people, then others will have the opportunity to read this poem. A poem that has many nominations on it doesn't really make sense to me and, IMO after two nominations the rest are just wasted and could be used to nominate other poems.

Not sure if this is a good idea...just makes sense to me and...I'm sure Janis would be able to 'fix'it so that...if someone tries to make a 3rd nomination...it will tell you no....go vote for something else haha!!! I think it could also stop judges being influenced by poems with masses of nominations on them.

1) Some addl. clarifications:
Combinations possible with current scoring scale:
3x5 =15 (where 3 is the 3 scoring options available, and 5 is # of judges)
Combinations possible with 1-5 scale:
5x5 =25 (not 3125 as in lotto!)

Makes ties much less likely, by a factor of 5:3 (reduces by almost half). Hopefully, this answers qns from Hellon and Sibs, whether the scheme really reduces the probability of ties.

2) Re: Say for example a judge thinks one poem stands out and gives it five. Wouldnt it be unfair on that poet and poem if the second best poem got voted the same amount? (Calm^)

The second best poem should then be voted a 4 (and not a 5, though it is available for use) by the same judge, if he want to ensure that his only 5 carries some weight. Whereas, if another judge feels 2 of the best poems to him/her are equally good, he/she would still have two 5's, and likewise two 4's, available to him/her for use, giving credence to point #2 (second post). The other judges then use the same analogy for their favorite's. But as all judge's vote on all poems (the most important factor in this scheme), they weigh in their opinion on those poems that is another judge's favorite/s, making it possible for that contestant to have a chance to win, if he/she is favored by a majority of the judges.

3) Re:Nana (or whoever) would somehow have to make sure that each judge was voting on each poem. (Sibs ^)

Should not be too difficult, for Janis (rather than the person counting). Currently, I believe you cannot vote twice on any poem and can use each score only once. The system points it out to you if you try to inadvertently do otherwise. You can simply request Janis, that the judge should get a warning that he has not scored on all poems. Say if he forgets 1 or 2, a warning in red pops up saying your scoring is incomplete. Also, if you plan to reveal the scores to the contestants, yes you have a safeguard, by having the tabulator verify that each poem has 5 scores against their name. If one or more is missing, you know someone forgot to score and the contest does not close till it is rectified.

4) Re:First, I dont think it would change anything with regard favouritism voting (Calm^).
Voting on all poems definitely dilutes favoritism/bias, unless a single judge votes a 1 or 2 on a really good poem voted 5's and 4's by his/her colleagues....then the mods should be able to pick the outlier, and over-ride the number by using a mean from the other judges or cite the judge, if there is a pattern. I agree it does not eliminate the human factor/bias completely, but it is still a big improvement on the current system.

Hope this satisfies some if not all of your qns. Yes, it will be a definite improvement over the current one. Merry X'mas!

I really need to pay more attention in class...do we only have three judges this time around....not 5?

Hellon, we have 5 judges.

Edit: Perhaps A Lonely Soul is suggesting we have 3 judges instead of 5? I haven't had a chance to fully grasp all of what is being said here, once I do i'll be back with an opinion.

Haven't read any of this, but I say if we find a way to make the judging system better, let's go for it. Janis is showing up these days, might as well take advantage of his presence.

Well...I'll try to rein my horse in until you do have time to read it all Mel....Jeez...Bugger...they should ban Christmas from every falling on a Sunday in Ozzz....Missed the Sunday Sesh...AGAIN LOL!

The "3" lonely soul was talking about was the three scoring options currently available (4, 7, 10). When comparing 3x5 to 5x5, the 5 point system certainly gives fewer ties. However, it's not taking into account the fact that a judge can use multiple fives, whereas they could only use one 10. It's how this factor would affect ties that I was wondering about. Maybe we can hire abby to run some stats... ; )

Edit: As a lonely soul pointed out to me earlier, restricting the number of votes for each value WOULD counter some of the likelihood of ties, but I'd still be interested in seeing some numbers, if anyone has the chops for some probability statistics. It's always been a weak point in my logical abilities!

So Nana is going to have to watch everything to make sure a poem got voted a one when someone said it did? That is a lot of work making sure people vote they way they say they did. And then those who have a few points will want to see comments as to why they were voted a 1 and not a 3, and bitch about how unfair it is. So then every judge has to comment every poem..? The weekly winner post will take hours to compose and read through.

It's hard enough finding judges as it is, you tell people they have to vote for every poem and get slack for lower votes, and then the constant complaining of no comments. I see judges dropping like flies and no one wanting to be a judge. You guys are way over thinking this and making it way too complex.

As far as nominations go, Hellon, I think your suggestion is kind of silly. It's like saying one poem can't be too popular. It people want to waste their nominations on a poem already nominated, then let them. Making it to we have to find other poems etc, you're going to get 100 poems on the noms list. We have some great poetry now nominated but there is also a lot of crap,.too. doing this and making the judges do all that extra work? I just flat would stop participating. This is all getting to be so ridiculous.

^ Not really Britt
RE: So Nana is going to have to watch everything to make sure a poem got voted a one when someone said it did?

This is easy for the programmer, please read #3 of my last comment above. The tabulator has to do essentially nothing more than what they are doing now.

RE:So then every judge has to comment every poem..? The weekly winner post will take hours to compose and read through

Nope. Only 2 or 4, to justify their 5's and 4's. Something they already do now. The weekly winner post will be now shorter, as there will be fewer if any ties. The HM's will also remain the same.

RE:As far as nominations go, Hellon, I think your suggestion is kind of silly.

Hellon has a valid point. Her soln to her concern is a little different than what I have. But I will let her get back to you on that first.

I agree with Britt.
You're making it way too complex.

and it doesn't matter if a poem gets 100 nominations.

It's not like I'm happy with the voting/judging system, but because I was a judge, I know how hard it is to go through all of the poems, and now... it became a lot harder because all the nominated poems appear on the list.

So... no complexity needed, whatsoever.

What about simply removing the weekly contest..

and let everyone enjoy themselves in their clubs or whatever..without worrying about the frontpage..

hahahaha

people will always complain anyways.

Nahhh.. the weekly contest is like one of the four legs of a chair.. if it's nt there, imagine what will happen...

whoever wants to complain, leave the contest alone! complain about the Logo, how it sucks.. the colors of the website... nana's inability of being a normal pnqer..........
but not the contest!

As far as nominations go, Hellon, I think your suggestion is kind of silly.

^^^^

It was only a suggestion not a solution and....really it's not any sillier that the suggestion you HAVEN'T made in this thread as yet.

Ok the voting system is certainly fallible, no doubt about that! having a tie and leaving it to the system to decide which poem will make the front page is wrong, either making both poems winners (or three if theres more than two) may the simplest way under the current system, or you change the whole box and dice. We have to read them all anyway wouldn't be to hard to rate each one, as per a Lonely Souls system but only comment on your top three as we do now

I have one other concern that is occurring now and that is each judge can see what poems have been scored, so if you are one of the last judges to vote you can manipulate your votes to make the poems you like get on the front page. Judges voting should be blind..!

Reminds me of one of the things I had to memorize on my US citizenship examination: the most important constitutional amendment, which makes this country great and different than most others and attracts immigrants from the world over:
The 1st Amendment (US constituton) :

The right to freedom of speech, press, religion, peaceful assembly and requesting change of government (in this case change of judging methods/rules).

I loved the questions and comments so far, to the point and constructive. So let it keep coming. PnQ changes should be ruled by a general consensus (not threats^), and certainly not by individual opinions to shut down a constructive debate. I am just a PnQ citizen, contributing to betterment, not a mod or anyone prominent, and certainly not a very good writer, by PnQ winning standards. But, if I can contribute to an impasse, between the old school and new ideas, I will try. So far, almost everyone agrees (-2), that PnQ judging methodology needs improvement, then why not? So let your constructive comments keep flowing. What do you think Sir Larry? (Aren't you a lawyer by Profession?)

I have made plenty of suggestions in these threads regarding the weekly contest, not only now but for the past year and a half. We are beating a dead horse.. people are only making it more complicated and we are going to end up having no one agree to judging. Unless you've been a judge in the past year you really don't know what all you do, and its not easy now. It eats up your personal time. Suggestions being made now are only making things ridiculously complicated. If the mods want to go back to being judges because we've run this thing into the ground with difficulties feel free. I'm so sick of this damn website and all the constant bitching. Can't anyone be happy for five seconds? Nothing is ever good enough.

I am happy \0/

I'm happy :D

Here are another 2 suggestions to add to the brew from my point of view:
1) Expand the weekly winners from 3 to 5, to keep up with the increasing number of very good poems in the nomination list, since the page no longer kicks out some deserving poems, and since there are a lot more members contributing good work. This will allow some lesser known (or newer) members to have a chance to be recognized...who does not want to be? for their labor of love for sharing their heart and soul with the rest of us. This way, the seniors/populars, etc will not feel that threatened, and the "juniors" will have more of a level chance.
2) There should be at least a 2-3 line comment to justify the nomination. This was my soln to Hellon's comment on which some others differed. This way you can not only reduce the number of members putting their names to the nomination (to make it weighty), but also curb the less qualified (nuance) nominations, which the judges feel were not appropriate (polite for trash...difficult for me to call anything trash...someone's labor of love, perhaps an Asperger citizen of PnQ). Providing a comment or good interpretation does help the judges making a decision, when they could have easily ignored a poem that they could not fathom in the first read.

Where did you come from....

I don't think you have to comment to nominate. Sometimes I nominate a poem and comment the next day or a week later, but I still feel it deserves the weekly. There are too many restrictive suggestions being thrown out there and were going to lose a lot of participation. It should be easier and more user friendly, not the opposite.

I'm listening

Gosh, I remember a few months ago me suggesting changing the judging system and being scorned, cursed, and humiliated...it was the start of my decision to delete..anyway..God bless you all..whatever you perfect people decide is fine with us peons...

Definite improvement - judges not being able to see the votes already cast, I think everybody agrees on that and Janis could fix it quite easily.

The rest of the suggestions are opinion and from what I can see, each have pros and cons. We could debate them until the cows come home and a locked thread later have not achieved anything.

I think we are forgetting how quite a few improvements have already been made to the contest recently and for the most part it works well overall. There is no perfect system that will keep everyone happy. While suggestions can be interesting I think we should remember to take this into account

I think we can discuss changing the judging system so that it more fully represents the judges' votes and intentions. We want to make sure that the "best" poems (according to the judges) are winning, and that the process is aided by the voting options.

Britt has a good point about keeping things in perspective, though. We need to keep in mind that 1) none of these large-scale changes can be made without Janis, and 2) In addition to fairness, we need to also be working to make things user-friendly and as streamlined as possible. The judges this year have a more strenuous job than judges of the past; and since we appreciate them greatly, let's not scare them away. They already read 35-plus poems per week, rank them, and write thoughtful comments for the top 3. For any changes that are made, we want to keep them in mind.

I would also, I guess, be against a rule requiring people to comment with their nominations. Nominations simply raise a poem for consideration. Comments are lovely, nominators don't necessarily have to defend their nominations. I don't always nominate the poems I think are the best - sometimes I nominate poems that I think are interesting or worthy of judges' consideration, but that don't have any nominations yet.

What if we were to simply change the voting options to 1-5, and judges still voted for their top 3 poems? Would that counteract some of the ties? Then, a judge might give a 5, a 4, and a 1, if they have two very good poems but nothing else stands out to them too much that week. Usually we don't have that issue, but it's happened in the past. Just throwing out yet another idea. I haven't thought it through too much yet.

"I have one other concern that is occurring now and that is each judge can see what poems have been scored, so if you are one of the last judges to vote you can manipulate your votes to make the poems you like get on the front page. Judges voting should be blind..!"

I definitely agree with this. Like, right now, we have moderators and then "super mods," with even more powers. If we could stratify the judges like that (so the contest runner and mods could see the votes) that would be a big step, I think.