PDA

View Full Version : JSON.decode a large JSON string takes forever



Xorbis
03-11-2018, 03:45 PM
I am trying to decode this JSON and it takes forever:



function onLoad()
WebRequest.get("https://swdestinydb.com/api/public/cards/", self, "cardsResponse")
end

function cardsResponse(response)
cards = JSON.decode(response.text)
end


Am I doing something wrong?

dzikakulka
03-11-2018, 05:29 PM
Well it is over 500k chars... JSON format bloats it horribly, it could be compressed to a fraction of that simply having codes
(instead of strings like "set_codes", "affiliation_code", "affiliation_name" et cetera repeated hundreds of times) and one dictionary to decode it later.

If you really want to use that API directly, refer to specific cards using a sublink like https://swdestinydb.com/api/public/card/01001 and decode them one by one in a coroutine.

Xorbis
03-11-2018, 08:00 PM
I was able to load each set of cards individually by first calling the sets/ endpoint in their API.
It does still however take a lot of time, about 1 minute to read all cards in all sets.

I have seen 100x faster JSON decoders in NodeJS, must have something to do with the LUA JSON module Tabletop Simulator is using.

dzikakulka
03-12-2018, 03:20 AM
Decoding json files in regular Lua console works waay faster for me too. As far as TTS goes, just slapping some JSON module in (did this one https://github.com/LuaDist/dkjson) speeded up decoding by ~20%. It's not the most lightweight one out these so figure some other could do even better. But all in all preprocessing those files before decoding should be able to reduce times A LOT anyway, I'd try that if it takes too long.

staticrealm
04-19-2019, 08:28 PM
I was able to load each set of cards individually by first calling the sets/ endpoint in their API.
It does still however take a lot of time, about 1 minute to read all cards in all sets.

I have seen 100x faster JSON decoders in NodeJS, must have something to do with the LUA JSON module Tabletop Simulator is using.

I just his this myself... what took about 15 seconds to parse in TTS.... in command line with jq its way fast. Really disappointing.

06:27:48 ~/dnd/strahd$ time cat durmacar.json | jq '.name'
"Durmacar"

real 0m0.246s
user 0m0.234s
sys 0m0.017s

staticrealm
04-20-2019, 12:33 PM
I just his this myself... what took about 15 seconds to parse in TTS.... in command line with jq its way fast. Really disappointing.

06:27:48 ~/dnd/strahd$ time cat durmacar.json | jq '.name'
"Durmacar"

real 0m0.246s
user 0m0.234s
sys 0m0.017s


I just went and pulled in code from https://github.com/rxi/json.lua/blob/master/json.lua and made it inline. Now it runs in a respectable manner.. this reallly needs to be fixed.

Sancho
07-21-2019, 04:46 PM
I just went and pulled in code from https://github.com/rxi/json.lua/blob/master/json.lua and made it inline. Now it runs in a respectable manner.. this reallly needs to be fixed.

This helped me fix the problem with decoding in my mod. Thanks! Worth noting that the encoder wasn't as forgiving as the one included with TTS. It can still be used, but might require tables to be more consistent (no spare arrays, no mixing of data types).

Supposedly this is the code used for JSON encoding: http://regex.info/code/JSON.lua

I noticed it uses a lot of table.sort calls to sort keys, which could be part of the reason why it takes so long.

It's worth noting that using JSON is not the only way to store your data. I had a long discussion with others on Discord and it was pointed out that any string can be stored and loaded in TTS. One suggestion was to store your data as lua and then use dynamic.eval to load the data (http://www.moonsharp.org/additions.html)

Personally, I find using the the decoder above is enough to make load times much more reasonable. Though I'm still looking for a way to more quickly encode without writing custom code for it.