Results 1 to 7 of 7

Thread: JSON.decode a large JSON string takes forever

  1. #1

    Exclamation JSON.decode a large JSON string takes forever

    I am trying to decode this JSON and it takes forever:

    Code:
    function onLoad()
      WebRequest.get("https://swdestinydb.com/api/public/cards/", self, "cardsResponse")
    end
    
    function cardsResponse(response)
      cards = JSON.decode(response.text)
    end
    Am I doing something wrong?

  2. #2
    Well it is over 500k chars... JSON format bloats it horribly, it could be compressed to a fraction of that simply having codes
    (instead of strings like "set_codes", "affiliation_code", "affiliation_name" et cetera repeated hundreds of times) and one dictionary to decode it later.

    If you really want to use that API directly, refer to specific cards using a sublink like https://swdestinydb.com/api/public/card/01001 and decode them one by one in a coroutine.

  3. #3
    I was able to load each set of cards individually by first calling the sets/ endpoint in their API.
    It does still however take a lot of time, about 1 minute to read all cards in all sets.

    I have seen 100x faster JSON decoders in NodeJS, must have something to do with the LUA JSON module Tabletop Simulator is using.

  4. #4
    Decoding json files in regular Lua console works waay faster for me too. As far as TTS goes, just slapping some JSON module in (did this one https://github.com/LuaDist/dkjson) speeded up decoding by ~20%. It's not the most lightweight one out these so figure some other could do even better. But all in all preprocessing those files before decoding should be able to reduce times A LOT anyway, I'd try that if it takes too long.

  5. #5

    JQ parse way faster... JSON PARSING SLOW WHY!?!?!

    Quote Originally Posted by Xorbis View Post
    I was able to load each set of cards individually by first calling the sets/ endpoint in their API.
    It does still however take a lot of time, about 1 minute to read all cards in all sets.

    I have seen 100x faster JSON decoders in NodeJS, must have something to do with the LUA JSON module Tabletop Simulator is using.
    I just his this myself... what took about 15 seconds to parse in TTS.... in command line with jq its way fast. Really disappointing.

    06:27:48 ~/dnd/strahd$ time cat durmacar.json | jq '.name'
    "Durmacar"

    real 0m0.246s
    user 0m0.234s
    sys 0m0.017s

  6. #6
    Quote Originally Posted by staticrealm View Post
    I just his this myself... what took about 15 seconds to parse in TTS.... in command line with jq its way fast. Really disappointing.

    06:27:48 ~/dnd/strahd$ time cat durmacar.json | jq '.name'
    "Durmacar"

    real 0m0.246s
    user 0m0.234s
    sys 0m0.017s

    I just went and pulled in code from https://github.com/rxi/json.lua/blob/master/json.lua and made it inline. Now it runs in a respectable manner.. this reallly needs to be fixed.

  7. #7
    Join Date
    Sep 2016
    Posts
    216
    Quote Originally Posted by staticrealm View Post
    I just went and pulled in code from https://github.com/rxi/json.lua/blob/master/json.lua and made it inline. Now it runs in a respectable manner.. this reallly needs to be fixed.
    This helped me fix the problem with decoding in my mod. Thanks! Worth noting that the encoder wasn't as forgiving as the one included with TTS. It can still be used, but might require tables to be more consistent (no spare arrays, no mixing of data types).

    Supposedly this is the code used for JSON encoding: http://regex.info/code/JSON.lua

    I noticed it uses a lot of table.sort calls to sort keys, which could be part of the reason why it takes so long.

    It's worth noting that using JSON is not the only way to store your data. I had a long discussion with others on Discord and it was pointed out that any string can be stored and loaded in TTS. One suggestion was to store your data as lua and then use dynamic.eval to load the data (http://www.moonsharp.org/additions.html)

    Personally, I find using the the decoder above is enough to make load times much more reasonable. Though I'm still looking for a way to more quickly encode without writing custom code for it.

Similar Threads

  1. [ADDED] JSON Object
    By Abarden in forum Scripting Suggestions
    Replies: 4
    Last Post: 05-08-2018, 02:33 AM
  2. [CONNECTION ISSUES] Server browser takes forever to load
    By nightfuryninja in forum Technical Support
    Replies: 1
    Last Post: 03-09-2018, 12:54 AM
  3. Read JSON from URL
    By FoaS in forum Scripting
    Replies: 0
    Last Post: 09-04-2017, 09:47 AM
  4. .json Error: Out of Memory
    By Eskander in forum Technical Support
    Replies: 2
    Last Post: 12-19-2016, 06:38 PM
  5. [SOLVED] json memory error message
    By Maven in forum Technical Support
    Replies: 3
    Last Post: 11-25-2016, 04:21 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •