Index
Tools For PGN Editing And Data Extraction
Norm Pollock Chess Downloads: Norm Pollock has written 66(!) utilities that can be used to edit and extract information from PGNs, EPDs, and regular text files. Among the things you can do with them is:
1) Clean up PGN tags
2) Add Elo ratings for White and Black
3) Truncate PGNs
4) Edit opcodes
5) Find specified material balances
6) Extract specified number of random text lines
Look at the readme files at Norm's site to see all the available utilities and their descriptions.
I have found Norm's utilities to be extremely valuable. I would be lost without them.
Back to Top
PGN Extract: A versatile command line program created by David J. Barnes for extracting data from PGN files. A few of the many things it can do are:
1)Remove exact duplicate games as well as removing games containing the same moves
2)Find games matching a given material balance
3)Finding games of a given variation
4)Finding games based on PGN tags
5)Add ECO codes
and much more.
For those who shy away from using the command line, Ferdinand Mosca created an interface for PGN Extract (Windows only) that is easy to use and covers much of the most useful functions of PGN Extract.
Back to Top
PGN Scanner: Much like PGN Extract, PGN Scanner (by Gabriel Guillory) is a command line program for extracting data from PGNs. It is possibly more versatile than PGN Extract, though not quite as robust. PGN Scanner can be used to create engine books, and accbook.dll can be incorporated in an engine so that it can use PGN Scanner books.
Back to Top
Scid (Shane's Chess Information Database): A chess database program (and more) created by Shane Hudson and maintained by him until 2004, and then maintained by Pascal Georges.
Among the things one can do with Scid are:
1) Analyze games
2) Annotate games
3) Filter games via header tags, material, or patterns
4) Add ECO codes
5) Remove duplicates
6) Tune Polyglot books
and much more.
There have been several developmental forks from this free and open source program:
ChessDB - maintained by Dr. David Kirby
Scid vs. PC - maintained by Steven Atkinson
Scidb - maintained by Gregor Cramer (only compatible with Linux at this time)
Scid960 - maintained by Ben Hague
Each fork has some aspect to it that is an improvement over the original program. ChessDB allows importing of multiple PGNs at one time and uses hypothesis testing (Chi squared test) to help determine if a particular move found in the tree window is better than other possible moves. Scid vs. PC adds the ability to conduct engine tournaments and improved Polyglot book tuning. Scid960 is able to handle chess 960 databases. I am not yet familiar with Scidb, but it seems to be able to open ChessBase databases (.cbh files) and handle chess960 and shuffle chess databases. I personally use ChessDB, but the best version for you depends on your needs.
Back to Top
José: A graphical database created by Peter Schäfer. José allows a user to edit games, add comments and variations, do positional searches, analysis games, create HTML and PDF files, etc... . It appears that work and support ceased in 2006. A fork, JoséChess 2 appears to also be dormant.
Back to Top
ChessX: A chess database program (inspired by Scid) maintained by Michal Rudolf. ChessX allows a user to edit and analyze analyze games. The last version was released in November, 2010, so I am uncertain if the project is still active.
Back to Top
Chess Query Language (CQL): A command line tool that allows more complex database searches to be made.Developed by Gady Costeff and Lewis Stiller. I found that it is not difficult to learn how to use this tool, especially since multiple .cql script samples are available to study. A graphical frontend, Visual CQL, is available (I have never tried it).
Back to Top
GUI Adapters
WB2UCI: A utility created by Odd Gunnar Malin that enables a native Winboard engine to be played in a GUI that only uses UCI commands, such as Shredder and Fritz.
To use Wb2Uci, you must place two files, Wb2Uci.eng and Wb2Uci.exe, in the same folder with the engine you wish to adapt to UCI. Wb2Uci.exe is the actual adapter, and Wb2Uci.eng is the text file that tells the adapter which engine to connect with and what engine options to send to the engine. Here is the contents of a simple Wb2Uci.eng file:
---------------------------------------------------
Name=Crafty 23.3 32-bit
Author=Robert Hyatt
Filename=Wb2Uci.exe
[OPTIONS]
Program=crafty-23.3-win32.exe
---------------------------------------------------
We see the name that will be sent to the GUI (Crafty 23.3 32-bit), the author's name, the adapter filename (which could be renamed to something such as crafty-23.3-win32_Uci.exe if desired), and the file name of the program the adapter is to connect with. While this will work, for some engines it may be necessary to include specific WB commands in the 'Options' section. For example, if I want Crafty 23.3 to use the Namilov tablebases stored on my computer, I would change the .eng file in the following manner:
---------------------------------------------------
Name=Crafty 23.3 32-bit
Author=Robert Hyatt
Filename=Wb2Uci.exe
[OPTIONS]
Program=crafty-23.3-win32.exe
NalimovPath=tbpath|string|c:\EGTB
NalimovCache=cache %iM|spin|8|1|32
---------------------------------------------------
In reality, the Crafty.rc file allows me to control the EGTB path, EGTB cache, book, etc... for Crafty. However, there are some engines that have to have the options entered into the .eng folder. Check out the Wb2Uci documentation and the Engines.txt file in the Wb2Uci download for more information.
Note: The executable that should be installed in the GUI is Wb2Uci.exe (or whatever you renamed it to). Remember, we want Wb2Uci to act as an interpreter, so the GUI must communicate directly to Wb2Uci.
There is a folder available at Le Système du Suisse containing oapproximately 300 .eng files (look for wbconf.zip near the bottom of the page).
Directions for using Wb2Uci to run Winboard engines in Fritz 7 can be found here. I am not completely sure if these directions will work for newer versions of Fritz GUIs. However, if you treat Wb2Uci.exe (or whatever you renamed it to) as a UCI engine, the typical method for installing an engine in your version of Fritz should work. The same goes for Shredder. Also, if you want to use an engine logo, install it in the engine folder and rename it 'Wb2Uci.bmp' (or whatever you renamed Wb2Uci.exe to) so that Fritz and Shredder will use the logo with the desired engine.
Back to Top
InBetween: A utility created by Odd Gunnar Malin that can monitor communications between an engine and a GUI, and can be used to translate\modify commands sent from the GUI to the engine or vice versa.
In some cases, the form of a command that a GUI sends is not quite the same as what the engine expects (or it may be the other way around). Or, you may want to convert a command, such as 'sd 5' ('go depth 5' for UCI) to 'sd 10' ('go depth 10'). Here are the steps necessary to use InBetween to accomplish this:
1) Place InBetween.exe (you may rename it if you wish) and InBetween.ini in your engine folder.
2) Edit the InBetween.ini file. By default, this is how it looks:
[InBetween] ; You can set program option here if its not possible in the server program ; ; The priority flag is if you want the server app. to run with lower priority. ; This could be when you want use an engine to analyze games/positions and ; you also want to use the computer to other work. If this is not set it will ; try to see what the client wants and set the server to this priority. ; Don't use the high option is you don't know what you do. ;Priority := low, normal or high ; ;CommandLine := c:\Program files\Engines\Crafty\wcrafty.exe ; ; The debug switch is if you want to see the command flow in a window when the ; server and client talks. 1 means viewpoint is on the client interface, 2 means ; viewpoint is on the server interface, and finaly 3 means that the viewpont is ; after the translation (input to client and input to server). ;Debug := 0, 1, 2 or 3 ; ; A try to pass any control signal. ;Ctrl := False or True ; ; The logfile is if you want that InBetween log all communications. ;Log := logfile.log [Client2Server] ; Set the translation of client command to server here. ; Format: ; clientword := serverword [Server2Client] ; Set the translation of server response to the client here. ; Format: ; serverword := clientword
To activate a command, remove the semicolon at the left margin.
Enter the engine path at 'Commandline'
Activate 'Log' and give the logfile a name to record communications between engine and GUI. Or, view communications via the console window by activating 'Debug'.
Client is the GUI. Server is the engine. To modify a signal/command, select the appropriate section, activate the appropriate line, and enter the existing signal/command and what it needs to be converted into.
Here is an example. I recently ran a test comparing Gaviota playing at different depths. In Arena, a user can create a tournament where each engine is limited to a user-defined depth. In order for the various copies of Gaviota to play at different depths, I used InBetween to convert the command 'sd [ply]' sent from Arena to the depth I wanted a particular copy to use. Here is one of my InBetween.ini files to make a copy of Gaviota send a move after search 4 plies even though Arena tells it to search 12 plies:
[InBetween]
; You can set program option here if its not possible in the server program
;
; The priority flag is if you want the server app. to run with lower priority.
; This could be when you want use an engine to analyze games/positions and
; you also want to use the computer to other work. If this is not set it will
; try to see what the client wants and set the server to this priority.
; Don't use the high option is you don't know what you do.
;Priority := low, normal or high
;
CommandLine := C:\gaviota-0.85.1-windows\Gaviota_0.85.1.exe
;
; The debug switch is if you want to see the command flow in a window when the
; server and client talks. 1 means viewpoint is on the client interface, 2 means
; viewpoint is on the server interface, and finaly 3 means that the viewpont is
; after the translation (input to client and input to server).
;Debug := 0, 1, 2 or 3
;
; A try to pass any control signal.
;Ctrl := False or True
;
; The logfile is if you want that InBetween log all communications.
Log := Gaviota_ply4.log
[Client2Server]
; Set the translation of client command to server here.
; Format:
sd 12 := sd 4
[Server2Client]
; Set the translation of server response to the client here.
; Format:
; serverword := clientword3) Set up InBetween.exe (or whatever you rename it to) in the GUI as you would have done with the engine executable.
Back to Top
Polyglot: A utility created by Fabien Letouzey which is used to run UCI engines under Winboard (for older versions of Winboard; the latest version of Winboard has a UCI2WB adapter built in).
Where Wb2Uci acts as an UCI engine, Polyglot acts as a Winboard/Xboard engine. It recieves the Winboard commands and converts them to UCI commands, which it then sends to the engine (and vice versa). For each engine that is to be used with Polyglot, a .ini file has to be written. The .ini consists of a Polyglot section, which contains commands/parameters for Polyglot, and an Engine section, which contains commands/parameters for the engine. Here is an example .ini file:
[PolyGlot] EngineName = Fruit 2.1 EngineDir = C:\Engines\Fruit EngineCommand = fruit.exe Book = true BookFile = fruit.bin ; same directory as INI file, use full path otherwise [Engine] Hash = 64
Polyglot can be invoked from the commandline, or by batch file (script). If I saved the above example as 'Fruit.ini' and located it in the same folder as Polyglot, I would then create a batch file named Fruit_2.1.bat with the following line:
polyglot[use the full file name].exe Fruit.ini
Note: I am not exactly certain how Polyglot was/is used with older versions of Winboard. The above description is unnecessary for newer versions when used with PSWBTM or the built-in tournament manager (newest version of Winboard). The description is necessary using Polyglot books with engines that can not natively use them.
Several people have contributed to Polyglot after Fabien, including Fonzy Bluemers and Michel van de Bergh. Michel's version is notable due to the fact that it can be used with GUIs that use only UCI commands. It is UCI to WB and UCI to UCI. This allows the use of Polyglot books in Fritz (read the Polyglot documentation for some instruction for building books; I hope to have a guide written in the somewhat near future).
Back to Top
Elo Rating Programs
Bayeselo: A utility written by Rémi Coulom that processes PGN files and produces Elo ratings of the players. I will not bore anyone with the details of the mathematic methods used (unless you want me to ;)). I will say that it makes use of the draw information and performance as Black and as White (unlike the original Elo model which ignores this information) to create possibly more accurate ratings.
Here is a short tutorial of the most important commands:
Place PGN and Bayeselo in same folder
Double click on Bayeselo
Type at the cursor: readpgn [your file name].pgn
Type: elo (this puts Bayeselo in the correct mode to compute ratings)
Type: mm 1 1 (this makes Bayeselo compute the ratings using the draw information
and performance as White. For PGNs where engines play each
position as White and as Black, use 'mm 0 1' as White advantage
should be neutralized. Using 'mm' makes Bayeselo use the
default values, which Rémi computed from WBEC games.)
Type: exactdist (This is where you select the algorithmn (there are four) for
computing the confidence intervals. I recommend 'exactdist' or
'covariance'. I use 'covariance', IPON and CCRL
use 'exactdist'. Given enough games, it does not really
matter.)
Type: offset [Average Elo] (The purpose of this is to set the range of numbers
of the Elo ratings. If 'offset' is not used, then
the average Elo will be used is 0. IPON uses the
command 'offset 2800 Deep Shredder 12' to set
Shredder's Elo to 2800)
Type: ratings (Displays the rating list in console window. To print out rating
list type 'ratings>[file name].txt. The text file will appear in
the same folder as Bayeselo.)
Back to Top
ELOStat: A utility created by Frank Schubert for computing Elo ratings. ELOStat comes as a stand-alone utility (at the link I provided, and also is found built into Arena. Since I suspect it is used more often in Arena, my description will be based on that assumption.
First, open the PGN with Arena. Click on the 'Elostat' button along the bottom of the database window. When you do, ELOStat automatically runs. If you prefer a differt start Elo value (the average Elo), abort the run and enter the new start value (near the bottom), then run ELOStat.
It may take ELOStat a while to finish computing large PGNs. When it does finish, click on the 'Rating.dat' tab to see the ratings, click on the 'Programs.dat' tab to see how each engine performed against each opponent, and click on the 'General.dat' tab to see information concerning the entire PGN of games. Also, the .dat files (which are text files in this case) can be saved to a folder of your choice using the 'Copy files to' button.
Back to Top
Ordo: A utility created by Miguel Ballicora for computing Elo ratings. It was adapted from a rating program Miguel wrote in the 1990s to rank collegiate volleyball teams (IIRC).
Ordo runs from the commandline. Also, you can write a batch file. Here are the list of commands:
usage: ordo [-OPTION] -h print this help -v print version number and exit -L display the license information -q quiet (no screen progress updates) -aset general rating average -p input file in .pgn format -c output file (comma separated value format) -o output file (text format), goes to the screen if not present -s # perform # simulations to calculate errors -e saves an error matrix, if -s was used
Here is a sample batch file for processing the PGN file Birds_and_Bugs.pgn, setting the average rating to 2750 Elo:
ordo-win32.exe -a 2750 -p Birds_and_Bugs.pgn -o rating.txt
Here is the output:
ENGINE: RATING POINTS PLAYED (%)
Quazar 0.4 32-bit: 2938.3 22.0 30 73.3%
Hiarcs 13.2: 2909.6 21.0 30 70.0%
Booot 5.1.0: 2895.8 20.5 30 68.3%
Hannibal 1.1 32-bit: 2895.8 20.5 30 68.3%
Gull 1.2 32-bit: 2855.8 19.0 30 63.3%
Spark 1.0 32-bit: 2842.8 18.5 30 61.7%
Umko 1.2 32-bit: 2842.8 18.5 30 61.7%
Gaviota 0.85.1 32-bit: 2767.3 15.5 30 51.7%
Nemo 1.01b 32-bit: 2754.8 15.0 30 50.0%
Texel 1.01 32-bit: 2717.2 13.5 30 45.0%
Tornado 4.88 32-bit: 2691.7 12.5 30 41.7%
Frenzee 3.5.19 32-bit: 2665.7 11.5 30 38.3%
Dirty 23Mar2012 32-bit: 2652.4 11.0 30 36.7%
BugChess2 1.9 32-bit: 2625.1 10.0 30 33.3%
Deuterium 11.02.29.107: 2550.8 7.5 30 25.0%
FruitFly 1.0: 2394.1 3.5 30 11.7%
Back to Top
Miscellaneous
Game Analyzer: A utility created by Thomas McBurney to find engine blunders and pivotal positions in PGNs by way of the engine evaluations. If the engine evaluation changes is equal to or more than the value set by a user, Game Analyzer will flag that position as "interesting". Also, the user can specify which moves to compare (setting 'Move Range' to -1 makes Game Analyzer compare the current move to the previous move. The user can also give a score to use as the upper limit to the comparisons (any position with an evaluation above this score will be ignored under the assumption the game is virtually decided).
Back to Top
Joined: A utility created by Andreas Stable that finds errors in PGNs and extract game and move data. For example, it can find the position in a PGN file of games that had the most legal moves available. It can determine how many promotions of each type occurs in a PGN, or how many castling moves occurred and of which type.The simplest way to use Joined is to create the following batch file (or enter this in the commandline):
----------------------------------------------------
joined.exe -q -v127 [your_file_name].pgn >> [new_file_name].pgn
pause
----------------------------------------------------
Make sure that Joined, your PGN, and the batch file are in the same folder. Joined will scan the input PGN, write to the output PGN (removing all commentary and evaluations and correcting the notation), and display the move and game data in the console window:
Note: Joined does not like Arena's practice of using '...' (3 dots) at the end of a game that was adjudicated. Use a text editor to remove all instances of '...' in the PGN before processing with Joined. Also, the name of the PGN can not have any spaces.
Back to Top