|
Rules and Guidelines
Last update: 13th July 2012 in Prizes and awards
Chatbot Battles are open to various kinds of chatterbots - Website Bots, Downloadable Bots, Chatroom Bots and Android apps - written in any programming language. Your bot only needs to be available so people can talk with it. Botmasters are invited to enter their bot and do battle with opponents from all over the world.
Rules for entering are:
1) Only English speaking chatterbots are allowed to enter.
2) A maximum of 3 (three) chatterbots per person but please don't just change the bot's name and resubmit it. Bear in mind rule 3 and if your bots battle each other and produce the same answers, neither will score a point.
3) Clones - you are free to enter any AIML, Personality Forge, MyCybertwin, INF.net bots and any other type of bot. However, if during a match, two bots produce exactly the same answer, neither will score a point for that question. This is to stop clones that have had no extra work carried out on them. The only exception to this rule is if the original bot (A.L.I.C.E., Ultra Hal etc) enters the contest. In this case, it will be awarded the point whereas the clone will score zero.
4) Accessibility - your bot must be easily accessible to the judges.
a) We can not accept bots that are only available on CD ROM. They have to be available to download on the
internet.
b) We can not accept bots that require the judges to download a 3rd party chat program such as Mirc. Most
instant messenger bots such as AOL are okay to enter.
c) We can not accept bots that require additional downloads of various programs that are not already installed on Windows XP unless they are relatively small and easy to install.
d) We can not accept bots that are not online 24/7. With judges being from all over the world and in different
time zones it's not possible to schedule a time for judging.
e) If you are running some type of blocking filter on your bot to prevent user abuse it must be removed during
judging or you must provide a method where the judge can re-establish contact instantly. If the judge can not complete the
conversation, your bot will score 0 points for that match. Seeing how a bot reacts to foul language or abuse may be a part of the judging
process.
f) In short you are responsible to provide the judges with a bot that is easy to access or install.
5) There must be a way for the judges to copy and paste the output
from each bot. This is especially important in speech-only bots. It is time consuming to judge any contest such as this and
helping the judges to accurately record the responses is appreciated.
6) Bots created on Android apps are permitted to enter. These apps must be either free to download or a copy of the paid version must be accessible to us, either without us having to pay or by reimbursing us for the cost. Unfortuately, we have no way of testing Iphone apps. In the case of a phone app, it is not necessary to provide a way to copy the output but the bot's responses must be visible on-screen and not just spoken.
Contest Calendar:
Deadline to Enter: June 1, 2012
League Stage: June 11, 2012 to July 7 2012
Knockout stages: July 9, 2012 to July 20, 2012
Announcement of Winners: July 2012
Prizes and Awards:
As well as the kudos of beating your peers as a prize and maybe a T-shirt, mousemat or similar gift. I have received a $100 donation from Decadent Vapours and also $100 from the ALICE AI Foundation. I will split this along with any other donations as follows:
1st place - 50 percent of prize fund
2nd place - 35 percent of prize fund
3rd place - 15 percent of prize fund
I have received a £50 donation from Daniel Burke to give to the first prize winner and Evil Limited has kindly donated £75 to the best 15 minute conversation.
If anybody is interested in sponsoring the contest or donating some prizes, either money or T-shirts and things like that, please contact me.
Judges:
The judges will be selected from the general public to determine the winners in the contest. None of the judges will have a
bot in the contest to ensure fairness. The judges will be totally impartial and fair to all entries regardless of the bot
type or the owner. The judges will each be allocated matches to oversee and will be free to ask the bots anything they wish.
If during a match, the bot asks the judge a question, the judge must answer it unless this goes to more than three questions
in a row. Such tactics will be classed as stalling and the judge is then free to proceed to the next question without
answering the bot.
Judging - League Matches:
A league match will either consist of a question and answer session or a free flowing conversation.
Question and answer match:
Each judge will pose the same five questions to the two competitors in each match and will score a point to the bot who in their opinion, gave the better answer for each question. Each question will be spelled and phrased exactly the same way to ensure that each bot gets an equal chance. At the end of the match, the points will be tallied up to find a winner. The maximum score in each match will be 5-0. All judge's decisions are final.
Free flowing conversation match:
The judges will talk with both bots in the match for 5 minutes each. Any and all subjects may be covered. The judge will allow himself to follow the lead of the bot when necessary and will not be deliberately obtuse with the bots. Judges will be aware of the current state of AI and will converse accordingly. After the end of the match, the judge will decide which of the two bots gave the more convincing conversation and mark each bot out of 5. The chatbots are NOT expected to believe they are human. We are looking for the ability to chat, not to try and fool the judges.
To avoid wasting the judge's time, if the judge feels that the bot is talking nonsense or repeating itself all the time and shows no signs of improving, they can (at their discretion) terminate the conversation.
Judging - Knockouts:
The knockout stages will be judged as above but with 10 questions to each bot. This will continue until the semi-finals when
there will be 15 questions and also a 15 minute free flowing conversation with the judge. The losing semi-finalists will play an extra match with 15 questions and a 15 minute free flowing conversation to determine 3rd and 4th
positions.
Questions:
1) The questions may contain typos to test the bot and internet slang terms like "lol" may also be used.
2) The questions will not be trivia like but reasonable in that an average person would be able to answer them. For example: "Who was Michael Jackson?" is good. "What is the population of Russia?" is bad.
3) As this is an international contest, the questions will not favour any particular country. For example asking a
question about a certain country that only the people of that country would know. An example of this is "Who is the President of Finland?".
4) Mathematical, logic, memory and reasoning questions may be employed. Examples of such questions are, "What is
6-2?", "Can I eat a building?", "I have a blue shirt. What colour is my shirt?", "Joe and Jim are twins. Joe is 24 years old,
how old is Jim?" and so on.
5) A question may just consist of a statement to see how a bot reacts in ordinary conversation rather than just being
a question/answering program. An example of these are, "I went to the movies yesterday.", "There is nothing on TV
tonight."
6) The questions will be posed in British English but if the judge deems a bot is having trouble understanding due to
spelling, he/she may, at their discretion, pose the question in American English. An example is, "What is your
favorite book?" instead of "What is your favourite book?"
7) Basically, anything goes. Each bot will be faced with the same input and so will face the same chance in each match as its opponent.
How the contest works:
Each entry in the competition is placed into a league table with at least three of the other entrants. The number of league
tables and entrants in each will depend on the number of contestants. A typical league table will look like this:
|
Group A
|
P
|
W
|
D
|
L
|
F
|
A
|
Pts
|
|
Chatty Chatbot
|
0
|
0
|
0
|
0
|
0
|
0
|
0
|
|
Testbot 123
|
0
|
0
|
0
|
0
|
0
|
0
|
0
|
|
Chatter Bot
|
0
|
0
|
0
|
0
|
0
|
0
|
0
|
|
Sir Talk-a-lot
|
0
|
0
|
0
|
0
|
0
|
0
|
0
|
|
Blah D Blah
|
0
|
0
|
0
|
0
|
0
|
0
|
0
|
Just like the soccer World Cup, each bot will play against another bot in its group until all the entrants in each group have
played each other. A match will consist of 5 questions. Both bots will be asked each of these questions with the one who
answers the best, in the judge's opinion, being awarded a point. If the answers are the same from each bot for
anything except a general knowledge or a yes/no question, as they may be for A.L.I.C.E clones for example, no points will be
awarded for that question. A typical match would go as follows:
Question 1: What is your name?
Bot 1: My name is Bot 1.
Bot 2: I do not know your name.
(Bot 1 scores a point for giving a correct answer. Score so far: 1-0)
Question 2: Do you like steak?
Bot 1: I like ice cream.
Bot 2: I like books.
(Bot 1 scores a point for giving a more relevant answer about food. Score so far: 2-0)
Question 3: What is a dog?
Bot 1: Dog: man's best friend, a canine.
Bot 2: Dog: man's best friend, a canine.
(No points scored. Although both bots were correct, they produced exactly the same response. The only exception to this rule would be if the original bot was taking part, as this is the original from which the clone was made. Score so far: 2-0)
Question 4: Jack is taller than Jim. Is Jim taller than Jack?
Bot 1: Who is that?
Bot 2: I do not know.
(Both bots got it wrong and so no points scored. Score so far: 2-0)
Question 5: What is the capital of France?
Bot 1: Paris.
Bot 2: Paris.
(Although they gave the same response, it was a general knowledge question and so both bots score a point. Final score:
3-1)
As in a soccer league table, 3 points are awarded for a win, 1 for a tie and 0 for a loss. The final result of the above
match was 3-1 to Bot 1 and so that entry would score 3 points for winning the match. Bot 2 lost and receives no
points.
These matches continue until everyone in each group has played each of the other members in the same group. The final table
would then look something like this:
|
Group A
|
P
|
W
|
D
|
L
|
F
|
A
|
Pts
|
|
Sir Talk-a-lot
|
4
|
3
|
0
|
1
|
14
|
3
|
9
|
|
Chatty Chatbot
|
4
|
2
|
1
|
1
|
10
|
8
|
7
|
|
Chatter Bot
|
4
|
2
|
0
|
2
|
13
|
9
|
6
|
|
Blah D Blah
|
4
|
1
|
2
|
2
|
8
|
12
|
5
|
|
Testbot 123
|
4
|
0
|
1
|
3
|
2
|
16
|
1
|
The letters along the top of the columns have the following meanings:
P = Played. How many battles the bot has played.
W = Won. How many battles the bot has won.
D = Drawn. How many battles that resulted in a draw/tie.
L = Lost. How many battles the bot has lost.
F = For. How many points the bot has scored.
A = Against. The number of points that have been scored against the bot.
Pts = Points. The total number of points (3 for a win, 1 for a draw, 0 for a loss)
So for example, let's say Chatty Chatbot has the following matches:
Chatty Chatbot (2) v (3) Sir Talk-a-lot
Chatty Chatbot (4) v (2) Testbot123
His entry in the group would look like this:
|
|
P
|
W
|
D
|
L
|
F
|
A
|
Pts
|
|
Chatty Chatbot
|
2
|
1
|
0
|
1
|
6
|
5
|
3
|
This is because he has played 2 battles, won 1, lost 1, scored 6 points in all his battles (2+4) and had 5 points scored against him in all his battles (3+2). He has been awarded 3 points as he has won 1 match.
The top two from each league table then go on to the knockout stages. The highest scoring runners up may also be
included in the knockout stages if the numbers are not a power of 2 (4,8,16,32 etc). In the event of a tie, the number of
points scored in all matches will be taken into consideration. If there is still a tie, the number of points conceded will be counted and if
by some reason there is still a tie after that, the tieing bots will play a match between themselves to decide a
winner.
The knockout stages continue in the same style as above but each match consists of 10 questions instead of 5 (or a conversation of 10 minutes instead of 5).
The winner will progress to the next knockout stage while the loser is out of the contest. In the event of a tie, further
questions will be asked until one bot falters. Once the contest gets down to the final four, each round will consist of 15 questions and each entrant will also have a 15 minute free flowing conversation with the judge. The knockout matches will follow this style:
|
Quarter Finals
(10 questions)
|
Semi Finals (15 questions +
15 minute conversation)
|
Final (15 questions +
15 minute conversation)
|
Contest winner
|
|
(1) Winner of group A
|
|
|
|
|
|
V
|
(9) Winner of match 1
|
|
|
|
|
(1) Runner up of group C
|
|
|
|
|
|
|
V
|
(13) Winner of match 9
|
|
|
|
(2) Winner of group B
|
|
|
|
|
|
V
|
(9) Winner of match 2
|
|
|
|
|
(2) Runner up of group D
|
|
|
|
|
|
|
|
V
|
(15) Winner of match 13
|
|
|
(3) Winner of group C
|
|
|
|
|
|
V
|
(10) Winner of match 3
|
|
|
|
|
(3) Runner up of group A
|
|
|
|
|
|
|
V
|
(13) Winner of match 10
|
|
|
|
(4) Winner of group D
|
|
|
|
|
|
V
|
(10) Winner of match 4
|
|
|
|
|
(4) Runner up of group B
|
|
|
|
|
There will also be a 3rd and 4th place play-off between the losing semi-finalists with 15 questions and a 15 minute free flowing conversation. The whole process should
take around 1 month depending on the number of contestants but this time frame may increase or decrease.
Note:
The rules and guidelines may be changed at anytime to ensure fairness. It is impossible to forsee all the events that may
arise during the contest so Chatbot Battles may have to make amendments to the rules to ensure the smooth running of the
contest.
|
|
