PDA

View Full Version : JSON.decode a large JSON string takes forever



Xorbis
03-11-2018, 03:45 PM
I am trying to decode this JSON and it takes forever:



function onLoad()
WebRequest.get("https://swdestinydb.com/api/public/cards/", self, "cardsResponse")
end

function cardsResponse(response)
cards = JSON.decode(response.text)
end


Am I doing something wrong?

dzikakulka
03-11-2018, 05:29 PM
Well it is over 500k chars... JSON format bloats it horribly, it could be compressed to a fraction of that simply having codes
(instead of strings like "set_codes", "affiliation_code", "affiliation_name" et cetera repeated hundreds of times) and one dictionary to decode it later.

If you really want to use that API directly, refer to specific cards using a sublink like https://swdestinydb.com/api/public/card/01001 and decode them one by one in a coroutine.

Xorbis
03-11-2018, 08:00 PM
I was able to load each set of cards individually by first calling the sets/ endpoint in their API.
It does still however take a lot of time, about 1 minute to read all cards in all sets.

I have seen 100x faster JSON decoders in NodeJS, must have something to do with the LUA JSON module Tabletop Simulator is using.

dzikakulka
03-12-2018, 03:20 AM
Decoding json files in regular Lua console works waay faster for me too. As far as TTS goes, just slapping some JSON module in (did this one https://github.com/LuaDist/dkjson) speeded up decoding by ~20%. It's not the most lightweight one out these so figure some other could do even better. But all in all preprocessing those files before decoding should be able to reduce times A LOT anyway, I'd try that if it takes too long.

staticrealm
04-19-2019, 08:28 PM
I was able to load each set of cards individually by first calling the sets/ endpoint in their API.
It does still however take a lot of time, about 1 minute to read all cards in all sets.

I have seen 100x faster JSON decoders in NodeJS, must have something to do with the LUA JSON module Tabletop Simulator is using.

I just his this myself... what took about 15 seconds to parse in TTS.... in command line with jq its way fast. Really disappointing.

06:27:48 ~/dnd/strahd$ time cat durmacar.json | jq '.name'
"Durmacar"

real 0m0.246s
user 0m0.234s
sys 0m0.017s

staticrealm
04-20-2019, 12:33 PM
I just his this myself... what took about 15 seconds to parse in TTS.... in command line with jq its way fast. Really disappointing.

06:27:48 ~/dnd/strahd$ time cat durmacar.json | jq '.name'
"Durmacar"

real 0m0.246s
user 0m0.234s
sys 0m0.017s


I just went and pulled in code from https://github.com/rxi/json.lua/blob/master/json.lua and made it inline. Now it runs in a respectable manner.. this reallly needs to be fixed.