TODO: Optimize Reconnect data

[evil-morfar]

The current reconnect data is horific: comm string example (68,829 kB) Decoded table.

Currently the following data is sent on reconnects: MLdb > council >candidates > lootTable > reconnectData.

The first two are required to get ML's settings, and to know if we have access to votingFrame (but we could initially just send the ML and the reconnectee if council). LootTable (as normally sent by ML) is required if we want candidate to roll - it's also currently required in the live version to setup votingFrame (not necessary if changed). ReconnectData should then contain everything that's happened in the session(s), but currently it's just a copy of addon.lootTable meaning it's a mixture of all of the above.

A session entry example

["equipLoc"] = "INVTYPE_FINGER",
["haveVoted"] = false,
["awarded"] = "зулналан-Ревущийфьорд",
["link"] = "|cffa335ee|Hitem:152063::::::::110:64::3:4:3610:42:1472:3528:::|h[Печать мастера порталов]|h|r",
["texture"] = 1391741,
["typeID"] = 4,
["subType"] = "Разное",
["relic"] = false,
["subTypeID"] = 0,
["lootSlot"] = 4,
["name"] = "Печать мастера порталов",
["classes"] = 4294967295,
["ilvl"] = 930,
["boe"] = false,
["quality"] = 4,
["candidates"] = { ...}

So what's needed? If LootTable is sent first, then basically everything but awarded can be removed. That's probably the way to go.
Otherwise the following are required for the votingFrame without needing to cache: awarded, link, ilvl. Most things from GetItemInfoInstant() should also be included when received.

A candidate example

["авилинна-Ревущийфьорд"] = {
        ["haveVoted"] = false,
        ["ilvl"] = 942.9375,
        ["class"] = "ROGUE",
        ["response"] = "PASS",
        ["gear2"] = "|cffa335ee|Hitem:133637:5429:::::::110:259::35:3:3418:1592:3337:::|h[Перстень короля Утгарда]|h|r",
        ["voters"] = {
         },
        ["role"] = "DAMAGER",
        ["diff"] = 0,
        ["votes"] = 0,
        ["specID"] = 259,
        ["isRelic"] = false,
        ["gear1"] = "|cffa335ee|Hitem:133634:5429:::::::110:259::35:3:3536:1582:3337:::|h[Кольцо из цепкого щупальца]|h|r",
        ["rank"] = "Четвертый",
},

Assuming candidates are sent first, all things from there can be removed (class, role, rank). SpecID could technically be included from candidate data, although it might not be entirely up to date. To further reduce data, everything that's false or 0 can be nil and recreated when received. Finally, we don't need entire itemlinks (we honestly never need to receive that on player gear, except for the loot history, where it's quite nice to have), so those could be shorted to itemstrings.

Solution

New structure (1 session):

reconnect = {
  ["авилинна-Ревущийфьорд"] = {
     ["ilvl"] = 942.9375,
     ["gear2"] = {
     },
     ["gear1"] = {
          [1] = 1,
     },
     ["response"] = {
          [1] = "p",
     },
  },
...
["lookup"] = {
   [1] = "item:137487:5890::::::::::35:3:3418:1597:3337:::",
},
}

• Get rid of the lootTable overhead - it's already contained in the actual lootTable comm. Awarded isn't required either, it can be gathered from the response.
• Get rid of all unrequired data from candidates.
• ilvl can be sent once instead of with every session.
• Don't send isRelic or isTier - while not 100% ensured, these are almost guaranteed if tier/relic buttons are enabled.
• Use lookup tables:
  • Default system responses can be shortened to one letter. Autopass = true saves even more.
  • Items only have to be written once. Each time that item appears, substitute with it's table index. This does add more data if there's no duplicate items, but saves a lot if there are.

Doing this changes the data from 68,829 kB to 15,368 kB, another example from 43,758 kB to 7,965 kB.

Further optimizing

• LibCompress. Using the above examples 15,368 kB becomes: 8,493 kB, 7,965 kB becomes: 4,549 kB.
• Shaving realmname (bit of a gamble, technically there could be two people with the same name): saves 729 kB and 317 kB in my examples.
• Reducing table key names: 527 kB and 491 kB.
• Substituting/reducing colons from itemstrings. Will probably be around 3 bytes per item. 543 kB and 246 kB.

[SafeteeWoW]

15018 Bytes are still too large to be use as reconnect data. I suggest to use LibCompress, and the original 68829 bytes are compressed into 34228 bytes by my testing. I believe 15018 bytes can be compressed to under 10k bytes, although this is still too large. The only is that it doesn't support backward compatibility.

---

The problem basically is caused by the bad data structure

--- Current:
lootTable[k].candidates[name].response
lootTable[k].candidates[name].ilvl
---Ideal:
candidates[name].response[k]
candidates[name].ilvl -- ilvl is the same for every session

The above solution first has one less table level. Second, it allow us to reuse some of the candidate table in core. Third, it's more serializing friendly, as the key to the table like "response", "ilvl", etc are not in the bottom level of the table.

[evil-morfar]

LibCompress is totally worth considering, however that probably won't be until v3, as it will break all comms. No matter what, a single comm containing all the data used will be large.

---

The structure is fine when handling a session (session is the identifier, everything related to a session is located in lootTable[session]), but indeed not optimal for transmission. Luckily, the data can be structured however we want.

1. It might have one less level, but you're also throwing away the data in lootTable - that would have to be sent either separately, or in another level, i.e. table = {candidates = {..}, lootTable = {data}}.
   Edit: Well, I suppose we wouldn't need that data if the lootTable is sent first.
2. There's no reusing in comms? Your suggestion would make each name and ilvl only show once instead once per session, a large improvement indeed. One further improvement, is use numbers instead of candidatenames as keys. That key would then be compared with the candidate table - only issue is to ensure the order in the candidate table.
3. Well, kinda.

[evil-morfar]

Using your suggested structure, albeit with a different table:
The current release code takes up 43,758 kB. Using this structure already reduces that to 18,336 kB Then I tried a few different things (each step contains the above):

1. Only send itemstrings: 11,328 kB.
2. Send response = true on autopasses: 10,480 kB
3. Further stripped unrequired data in itemstrings: 9,708 kB
4. Use a replacement table for responses (including autopass): 9,703 kB.
5. Autopass = true and replacement table: 9,597 kB.
6. Doing a lookup table for items so each item's only sent once: 7,965 kB.
7. LibCompres'ing all that: 4,549 kB.
8. Compressing and encoding: 4,639 kB.

I was interested in if LibCompress was more efficient in handling an item lookup table, so I tried compressing at step 5: 5,244 kB and encoded: 5,319 kB. Either it doesn't work as I expected, or it's just better to do it yourself.

The table looks like this when extracted.

For comparison, the original example goes to:
Serialized: 15,368 kB
Compressed: 8,493 kB
Encoded: 8,660 kB

[SafeteeWoW]

Using the data you posted: https://gist.github.com/evil-morfar/7cb190de8b9cce72139d7af07b8c1dfc

1. Compressed and encoded: 4634 Bytes
2. Removed "item:": 4343 Bytes
3. Then remove trailing colons: 4504 Bytes (larger)
4. Use string replacement so "ilvl", "response", "gear1", "gear2" are reduced to 1 byte: 3719 Bytes.
5. Serializer is clever and don't serialize table key when table is an array (replace "%^N"..num (num=1-6) by ""): 2938 Bytes
6. Reduce ^N, ^T, ^B, ^S, ^b, ^t, ^F to one byte: 2801 Bytes

[SafeteeWoW]

Tested on the RCSerializer which I completed early:

1. Remove "item:" and trailing colons
2. Serialized by RCSerializer
3. Compressed by LibCompress
   The result is 2863 bytes.

If Step 2 is changed to use AceSerializer, the result is 4202 bytes.
RCSerializer gives 32% smaller data.

[SafeteeWoW]

More optimization can be done:

1. Use GUID instead of player name. The realm id in the GUID is ignored when it is the same as the realm of ML. (There is some problem if ML changes when the reconnect data is being transmitted, but I don't think this really happens)
   I use the following code to test it. (

-- Assuming everyone is in the same realm of ML
		local function GenerateRandomID()
			local str = ""
			for i = 1, 8 do
				local r = math.random(0, 15)
				if r < 10 then
					str = str..strchar(r+48)
				else
					str = str..strchar(r+65-10)
				end
			end
			return str
		end

		for k, v in pairs(reconnect) do
			if type(k)=="string" and k:find("-") then
				reconnect[GenerateRandomID()] = reconnect[k]
				reconnect[k] = nil
			end
		end

My result is 2583 bytes

2. More aggressively, we can use raid id as table index, but we would have to invalidate reconnect data if group roster changes when it is being transmitted. (We update group members right before sending reconnect data)

Then reconnect data will only take 2160 bytes

3. I don't think the average item level of a player matters in the Loot Council. Loot Council usually looks at the item level difference of the item being decided. I personally never looks at the average item level of the player. I would like to move this to EU.

Combined with 2, and then remove ilvl data
1982 bytes.

[evil-morfar]

Keep in mind this is intended for v2.8.

I don't think it's reliable/fast to use GUIDs - currently we would need to fetch them all before being able to use the data. To use that, each client should maintain a GUID -> display name table, which in turn could be used by the candidate table.

I seem to remember seeing somewhere that raidid is not an reliable identifier across multiple clients.

I think the average ilvl is one of the most important information in there. It's the only useful statistic for a guild trying to gear up their members equally. Eitherway, removing core features in a v2.x update is out of the question.

[SafeteeWoW]

There isn't much room for optimization, if we want to preserve backward compatibility in v2.8.
I think replace item link with item string will make enough improvement in v2.8.

[evil-morfar]

It doesn't really make sense to preserve backwards compatibility, as there's not really any gain from it. My point was all the required resources (such as proper GUID handling) won't necessarily be ready for 2.8, and I'm not sure doing a major comms overhaul is the right thing to do either at this point.

For now, I only intent reconnect to break backwards compatibility in v2.8.