If you edit Module:Icon but don't care about our FFXIV gameplay coverage or Lua scripting, you should read the bullets at the end of this post. If you do care about FFXIV gameplay or our codebase, you should read this whole post. If you don't know what any of that mean, you should just ignore this post.
As pointed out by Adonzo, {{icon}} crashes after N uses in a single page for some rather small N (some experimentation indicates N = 197). At this point you get the annoying red text "Lua error: Out of memory" or whatever. In addition, if {{icon}} is used enough times in one page (I didn't bother to find an exact number here) the page loads much slower. The reason is that Module:Icon loads every single database for every single game listed in it. This was fine until we added coverage of FFXIV, which has 4,000 quests and 60,000 items, and growing all the time, which is why we're now getting these glitches.
There are two ways to load a database into Lua:
- mw.loadData, which preserves the database in memory until the whole page is written, or
- require, which forgets the database after the invoke completes.
I tried rewriting the module so that it only loads the database that it needs. This fixed all the issues except for pages that need to invoke the FFXIV Item database. (We now have N > 2880, for example; I couldn't test any larger than that because the sandsea editor crashed when I tried N much larger than 3000.) The reason is that for every database except FFXIV Item, we can mw.loadData; but if you try to mw.loadData FFXIV Item, then the module immediately crashes because we run out of memory. So that database must be required, at great performance cost: it comes in several modules, each of which must be recalled every single time you use {{icon}} to look up an item. (With the changes that I wrote in the sandbox, it will no longer be necessary to require FFXIV Item when you look up anything other than an item from FFXIV, as it should be.)
(Side note: why does FFXIV Item come in multiple modules? A MediaWiki plaintext file, which includes Lua modules, can be at most 2 megabytes long. FFXIV Item is larger than that, I think about 6 megabytes. One optimization would be to minimize the number of modules FFXIV Item is broken up into by collapsing it into a smaller set of modules, but I don't think this would give us much gain in terms of runtime, and the gains we'd get would decrease every time FFXIV is patched. I think JBed and I tried a similar trick some years ago by minimizing whitespace in some data modules, and it sort of worked, at the price of making those modules unreadable to a human, or indeed to anyone except the Lua compiler.)
I consulted devwiki about these issues, and the official response from a FANDOM staffer is that we should not use Lua for something that should be done with Semantic MediaWiki because Lua has such severe memory constraints, but also that we cannot add Semantic MediaWiki to FFWiki since we weren't grandfathered into having it whenever they stopped giving it out, and that they are not at liberty to say when FANDOM will start adding semantic-like functionality to wikis again.
Anyways, FFXIV Item is going to be required for the time being. This will cause XIV item pages to load rather slowly after any purge, but I don't feel too bad about this because those pages are several patches out of date already, just because they are nigh impossible for any human to maintain and furthermore because I don't have any better ideas beyond the slight gains we could get from collapsing the FFXIV Item database and by nuking whitespace.
(Side note: why are FFXIV item pages so badly in disrepair? It's because SCM, Adonzo, and I have all tried to maintain them, and all given up. Please love yourself and don't try to prove us all wrong.)
So that brings us to the issue of XIVAPI integration for FFXIV Item. Some time in June or July I will have enough free time to write the small mountain of Python needed to do this. Once this is done, I propose to structure our coverage of XIV items as follows: all the information will be stored in modules of the form Module:XIVAPI/Items/xyz where xyz ranges over three-character strings, and an item is put in module xyz if the first three letters of its lowercased name is xyz. More concretely, if we get a raid called "Eden's Flying Circus (Savage)" which drops the "Oracle's Arm", "Gaia's Bigass Hammer", it will be stored in Module:XIVAPI/Items/gai. The advantage of this formatting is that it's really easy for Lua to find Gaia's Bigass Hammer without knowing anything about it a priori, while more obvious organizational schemes, like sorting by location or item type, requires Lua to either search through several megabytes of data, causing all the performance issues I have detailed above, or for the average FFWiki to input a bunch of lookup data every thing they want to invoke information about Gaia's Bigass Hammer, or any other item for that matter. Module:XIVAPI/Items will be generated by Intangir Bot, using data from both XIVAPI and our current Icon database that was created by Adonzo.
Now, this still has serious performance limitations, so that we will not able to call upon Module:XIVAPI hundreds of times per page. In particular, even after some optimizations I have in mind (such as invoking icons directly from Module:XIVAPI/Items rather than going through a separate database), we will have to have fairly short item pages. However, this is not much loss, because I don't think most readers want to suffer through a several hundred item long page anyways, and besides Intangir Bot can just write these pages to be quite short. They will just look like "Final Fantasy XIV items/Oracle's Arm 42#Gaia's Bigass Hammer". (Nobody has to memorize the link to this, though -- we will still have icon functionality, and without the performance issues this time. Intangir Bot could even make redirects if we really wanted.)
I welcome any criticism of the above proposal. As for the changes to Module:Icon, I will make them live tomorrow after I get home from work, which is probably around midnight UTC, though I might have some drinks with friends (over Zoom; damn you COVID-19) so possibly later than that.
Now, how do the changes to Module:Icon affect functionality? You should have Module:Icon/sandbox open while reading the below.
- Instead of making game pages like Module:Icon/data/WoFF, you should update the table
local GAMESin the--- CONSTANTS ---section of Module:Icon to include a line["woff"] = "Module:Icon/data/WoFF/",(note the slash at the end, and the lowercase). - All data should be in submodules, say Module:Icon/data/WoFF/Element, and not in Module:Icon/data/WoFF itself.
- All modules, say Module:Icon/data/WoFF/Intervention Quest, should adhere to the capitalization convention That The First Letter Of Every Word Is Capitalized And No Others Are. (I will correct this.)
- All modules should have a name in MediaWiki that is (with any capitalization convention) identical to the name of their respective module. So you should write
{{Icon|WoFF|intervention quest|billybob's wild ride}}or{{Icon|WoFF|InterVENTION QuESt|billybob's wild ride}}rather than{{Icon|WoFF|interventionquest|billybob's wild ride}}. (I'll run Intangir Bot to correct this.) - If you are making a database for a game with a very large database (hi, FFXIV) and you need to make performance optimizations, or you want Module:Icon to be able to call some other database for it (also hi, FFXIV), you can now write a loader for it. A loader is a pure Lua function that eats a pair (category, icon) and returns the icon from the relevant category. In this case, you would update the table
local GAMESto say["ffxiv"] = LOADER_FLAG,and then write a loader in the section--- LOADERS ---, which should start withloaderz["ffxiv"] = function(category, icon)(i.e. it should be a lambda fed into the tableloaderz). If this sounds like gobbledygook to you, you can also just ask me to do it.