Live statistics chess960 from chess.com?

Question

Cross-posted chess se, but nothing.

Both lichess and chess.com have the feature to play the variant chess960 live. However, only lichess has a graph showing how your live chess960 rating has changed over time. Lichess also shows other statistics like highest, lowest, best wins, worst losses, average opponent rating, etc. (chess.com does have this for correspondence chess960 though.)

I could create my own graph and statistics in Excel/Google Sheets by manually recording each game's date and my rating afterwards indicated beside my username, but...

Question: Is there a way to obtain, or what in general is the way to go about obtaining, ratings after each chess960 game using some kind of script that sees a player's public profile and then extracts the data?

I have a feeling this kind of script has been done before even if this was not specifically done for chess.com's live chess960. The script doesn't have to graph (pretty easy to do once you have to the data: just use excel/google sheets). I just need the script to collect all the dates and rating numbers for each line of the user's games.

Edit 1: Not sure of on-topic, off-topic stuff on stack of, but i've posted on stack of before. My 1st post was in 2014. It seems these post is getting negative reaction due to that I seem like I'm asking to be spoon fed or something. I don't believe spoon feeding is necessarily an issue here if it's not some homework thing, and spoon feeding is not necessarily what I am asking or at least intend (or 'am intending' ?) to ask anyway. You could just give me the general ideas. For example, if this is to do with 'scraping' or something, then just say so.'

However, I don't quite see this question as any different as like these:

How do I get notified if SE tweets my question? --> Here you could argue I'm asking about se itself on se, so it should be allowed. I've asked chess.com people, but they haven't replied to me, so here I am.
Pricing when arbitrage is possible through Negative Probabilities or something else --> I mean is the guy spoonfeeding or whatever by writing the script?

Edit 2: Additionally, what I'm trying to get at in this post is avoiding the concept of reinventing the wheel/wheel reinvention. I mean, I can't possibly be the 1st person in the history of the internet to ever want to extract their data from chess.com or lichess or something. Plus, chess is a game that has been around for awhile. It isn't like csgo or valorant w/c is relatively new. I really do not see any point A - to look up myself how to do go about extracting data from a site as an alternative to manually typing it up myself and of course B - to manually type it up myself when it would seem pretty weird if there weren't already readily available methods to do this.

Update 2: Fixed now. see the 'json' vs the 'preformed'. WOW.

Update 1: It appears Mike Steelson has an answer here, where the code is given as

=arrayformula( regexextract(split( substitute(substitute(substitute(getDataJSON(A1;"/games";"/pgn");"[";"");"]";"");"""";"") ;char(10));"\s.*") )

with an example given here

https://docs.google.com/spreadsheets/d/1MX1o5qdy0K3gTMzbimUV3SmFf-0XPCSJ8Vz4IjI-8Ak/copy

It appears there's a problem when it gets to the case of chess960 only. Consider for example this player: Replacing 'gmwso' with the player's username will yield a weird output. i imagine the output will be messier for mixed chess960 and chess.

Welcome to Stack Overflow. Please take the [tour] to learn how Stack Overflow works and read [ask] on how to improve the quality of your question. Then check the [help/on-topic] to see what questions you can ask. Please see: [Why is “Is it possible to…” a poorly worded question?](https://softwareengineering.meta.stackexchange.com/q/7273) (don't just change your question to "How"). — Progman, Apr 23 '21 at 22:12
@Progman 'Assume that the answer is 'yes' and start designing the software. When you have a problem, then ask about that problem.' --> 1stly this isn't stackoverflow meta. it's software engineering meta. anyway, i'm not really asking someone to do it for me necessarily. i have a feeling someone must've done something like this already. probably for lichess or something. i'm just wondering how to do this for chess.com. edited question. — BCLC, Apr 24 '21 at 13:57
Please avoid cross posting the same question on multiple sites, see https://meta.stackexchange.com/questions/64068/is-cross-posting-a-question-on-multiple-stack-exchange-sites-permitted-if-the-qu — Progman, May 01 '21 at 11:31

Sleafar · Accepted Answer · 2021-05-01T13:44:57.797

1

Copy of my answer on Chess.SE, in case someone is looking here for an answer.

Yes, it's possible to obtain the data you want. Chess.com has a REST API which is described in the following news post:

https://www.chess.com/news/view/published-data-api

You can use the following URL to get a list of monthly archives of a players games:

https://api.chess.com/pub/player/{username}/games/archives

In this list you will find URLs which look like this:

https://api.chess.com/pub/player/{username}/games/{YYYY}/{MM}

If you append /pgn to this URL, you will get get all the games in PGN format, which is probably simpler to parse.

Lets look at an example:

https://api.chess.com/pub/player/magnuscarlsen/games/2018/01/pgn

Here you will find games played by Magnus Carlsen in January 2018. This list contains a couple of Chess960 games, which are identified by the following tag:

[Variant "Chess960"]

The following tags will give you the UTC date and time of the game as well as the players ratings at the time:

[UTCDate "2018.01.03"]
[UTCTime "21:50:55"]
[WhiteElo "2706"]
[BlackElo "2940"]

Lichess has also an API to download games, which I already described here.

Code

Here's some simple Kotlin code to extract the data (you will need to change the file and user name):

import java.io.File

fun main() {
    val data = (File("ChessCom_magnuscarlsen_201801.pgn").readText().trim().split("\n\n\n").map {
        it.split('\n').filter { it.startsWith('[') }.map {
            val t = it.replace(Regex("[\\[\\]]"), "").split(' ', limit = 2)
            t[0] to t[1]
        }.toMap()
    })
    data.forEach {
        if (it["Variant"] == "\"Chess960\"") {
            println("${it["UTCDate"]} ${it["UTCTime"]} ${it[if (it["White"] == "\"MagnusCarlsen\"") "WhiteElo" else "BlackElo"]}")
        }
    }
}

Result:

"2018.01.03" "21:50:55" "2706"
"2018.01.03" "21:09:41" "2727"
"2018.01.03" "19:43:22" "2703"

edited May 01 '21 at 13:44

answered May 01 '21 at 10:06

Sleafar

1,549
7
10

3 hrs until bounty expired. nice. thanks Sleafar – BCLC May 01 '21 at 10:21
' in case someone is looking here for an answer.' --> sounds like the most ridiculously humble thing i've ever heard. really thanks again. God bless you. – BCLC May 01 '21 at 10:22
btw feel free to repost answer here: [board games](https://boardgames.stackexchange.com/questions/54692/live-statistics-chess960-from-chess-com), [webapps](https://webapps.stackexchange.com/questions/154035/live-statistics-chess960-from-chess-com), [superuser](https://superuser.com/questions/1645900/live-statistics-chess960-from-chess-com) – BCLC May 01 '21 at 10:38
anyway, Sleafar, how does one convert this stuff into a spreadsheet? sounds like some really tedious like say python thing (python is the only language i learned where i know of a way to go about doing something like this) where there's some script that extracts every thingy beside 'UTCDate' and 'WhiteElo'. is there perhaps a wheel already invented for this? – BCLC May 01 '21 at 10:44
(additionally there's a hassle that there doesn't seem to be an immediate way to extract say the magnus carlsen elo. like you'd have to extract both white and black elo and then later in the spreadsheet or another script further sort out which one is carlsen, but never mind this) – BCLC May 01 '21 at 10:44
2

@BCLC I'm not aware that such a script exists. After all, the word "script" implies for me something that someone has created to solve his particular problem. And yes, you would need to parse the data and convert it to a format which you can import into a spreadsheet. – Sleafar May 01 '21 at 10:51
(ok i'm obviously not really into programming/computing/machine learning/whatever, so obviously i wouldn't really know the terms.) – BCLC May 01 '21 at 10:54
how do i parse the data please? i mean, is there really not some kind of built in way to export/parse/import/extract/convert certain data from chess.com or in general from a pgn file into a sheet? – BCLC May 01 '21 at 10:54
1

@BCLC That's the reason I usually don't post here anymore. Anyway, I added a Kotlin snippet to extract the data. You will probably need [IDEA](https://www.jetbrains.com/idea/) to use it. – Sleafar May 01 '21 at 11:40
You may answer here: https://stackoverflow.com/questions/67345784 – BCLC May 01 '21 at 11:41
Additional: OH WOW you even put specifically chess960 THANKS A MILLION – BCLC May 01 '21 at 11:46
Sleafar, i'm not so familiar with Kotlin, but i looked it up, installed the stuff, etc. 1 - i kinda forgot my python and R already, but do i need to download something for the 'import java.io.File' ? 2 - if so, then where do i put it? 3 - do i really just say 'ChessCom_magnuscarlsen_201801.pgn' instead of full directory? 4 - if not, then how do i specify a directory? or does kotlin search the entire computer for a file with this name? – BCLC May 01 '21 at 12:11
1

@BCLC You don't need any external libraries, and you will need to specify the full path to the file you downloaded. If you are using Windows you need to escape the path separator, i.e.: "C:\\path\\to\\the\\file.pgn" – Sleafar May 01 '21 at 12:17
Sleafar, any opinion of Mike Steelson's answer [here](https://stackoverflow.com/a/67346298) please? looks direct from the site to google sheets, but it appears the output gets messy when the games are chess960 only vs chess only. i imagine the output will be messier for mixed chess960 and chess – BCLC May 01 '21 at 12:48
An illegal reflective access operation has occurred --> what to do please Sleafar? (i mean to ask if this is the point where i take it from here eg [i should look up this myself](https://stackoverflow.com/questions/50251798/what-is-an-illegal-reflective-access) or if there's something wrong with your script) – BCLC May 01 '21 at 13:50
1

Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/231822/discussion-between-sleafar-and-bclc). – Sleafar May 01 '21 at 13:56

BCLC · Answer 2 · 2021-05-02T09:13:56.050

Ok I figured out a way to do Sleafar's output with only notepad and command prompt (and java/jar)

This is based on Run Kotlin in Windows cmd (and also this How to Download and Install Kotlin on Windows, but maybe all the stuff is already in the video):

Create a .kt (not .kts!) file for example extractgames.kt in for example C:\Users\user (see (A) below)
Open up cmd (command prompt) and then if need be run cd "C:\Users\user" to change directory.
Run kotlinc extractgames.kt -include-runtime -d extractgames.jar
Run java -jar extractgames.jar

(A) Re (1) above, i guess the code looks like

import java.io.File

fun main() {
    val data = (File("[insert full directory here and remove rectangular brackets. you may or may not need the \\ instead of just \]").readText().trim().split("\n\n\n").map {
        it.split('\n').filter { it.startsWith('[') }.map {
            val t = it.replace(Regex("[\\[\\]]"), "").split(' ', limit = 2)
            t[0] to t[1]
        }.toMap()
    })
    data.forEach {
        if (it["Variant"] == "\"Chess960\"") {
            println("${it["UTCDate"]} ${it["UTCTime"]} ${it[if (it["White"] == "\"[insert username here and remove rectangular brackets. i guess need not be case sensitive. but check for yourself]\"") "WhiteElo" else "BlackElo"]}")
        }
    }
}