Results 1 to 6 of 6

Thread: JSON.decode a large JSON string takes forever

  1. #1

    Exclamation JSON.decode a large JSON string takes forever

    I am trying to decode this JSON and it takes forever:

    Code:
    function onLoad()
      WebRequest.get("https://swdestinydb.com/api/public/cards/", self, "cardsResponse")
    end
    
    function cardsResponse(response)
      cards = JSON.decode(response.text)
    end
    Am I doing something wrong?

  2. #2
    Well it is over 500k chars... JSON format bloats it horribly, it could be compressed to a fraction of that simply having codes
    (instead of strings like "set_codes", "affiliation_code", "affiliation_name" et cetera repeated hundreds of times) and one dictionary to decode it later.

    If you really want to use that API directly, refer to specific cards using a sublink like https://swdestinydb.com/api/public/card/01001 and decode them one by one in a coroutine.

  3. #3
    I was able to load each set of cards individually by first calling the sets/ endpoint in their API.
    It does still however take a lot of time, about 1 minute to read all cards in all sets.

    I have seen 100x faster JSON decoders in NodeJS, must have something to do with the LUA JSON module Tabletop Simulator is using.

  4. #4
    Decoding json files in regular Lua console works waay faster for me too. As far as TTS goes, just slapping some JSON module in (did this one https://github.com/LuaDist/dkjson) speeded up decoding by ~20%. It's not the most lightweight one out these so figure some other could do even better. But all in all preprocessing those files before decoding should be able to reduce times A LOT anyway, I'd try that if it takes too long.

  5. #5

    JQ parse way faster... JSON PARSING SLOW WHY!?!?!

    Quote Originally Posted by Xorbis View Post
    I was able to load each set of cards individually by first calling the sets/ endpoint in their API.
    It does still however take a lot of time, about 1 minute to read all cards in all sets.

    I have seen 100x faster JSON decoders in NodeJS, must have something to do with the LUA JSON module Tabletop Simulator is using.
    I just his this myself... what took about 15 seconds to parse in TTS.... in command line with jq its way fast. Really disappointing.

    06:27:48 ~/dnd/strahd$ time cat durmacar.json | jq '.name'
    "Durmacar"

    real 0m0.246s
    user 0m0.234s
    sys 0m0.017s

  6. #6
    Quote Originally Posted by staticrealm View Post
    I just his this myself... what took about 15 seconds to parse in TTS.... in command line with jq its way fast. Really disappointing.

    06:27:48 ~/dnd/strahd$ time cat durmacar.json | jq '.name'
    "Durmacar"

    real 0m0.246s
    user 0m0.234s
    sys 0m0.017s

    I just went and pulled in code from https://github.com/rxi/json.lua/blob/master/json.lua and made it inline. Now it runs in a respectable manner.. this reallly needs to be fixed.

Similar Threads

  1. [ADDED] JSON Object
    By Abarden in forum Scripting Suggestions
    Replies: 4
    Last Post: 05-08-2018, 02:33 AM
  2. [CONNECTION ISSUES] Server browser takes forever to load
    By nightfuryninja in forum Technical Support
    Replies: 1
    Last Post: 03-09-2018, 12:54 AM
  3. Read JSON from URL
    By FoaS in forum Scripting
    Replies: 0
    Last Post: 09-04-2017, 09:47 AM
  4. .json Error: Out of Memory
    By Eskander in forum Technical Support
    Replies: 2
    Last Post: 12-19-2016, 06:38 PM
  5. [SOLVED] json memory error message
    By Maven in forum Technical Support
    Replies: 3
    Last Post: 11-25-2016, 04:21 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •