Friends, hello! More huge progress this fortnight. This is already a gigantic energy, so rather than spend time on the intro, let’s get into it:
Building out the dictionary
To begin with, in this fortnight I decided I had the objective of – by the end of the fortnight – being “done” for 0.11 with shifting as many generators as possible into the massive generator-information dictionary I’ve been slowly constructing across the past month (along with the also vast dictionary of dictionaries for everything within that dictionary, allowing the information from dozens of generators to be “standardized” when used for the generation of a clue, regardless of what characters make up the grids that constitute that generator, and regardless of what special rules might apply to the generation system in question). This is one of those tasks which puts a lot of things “up in the air” and for the sake of my brain and the sake of focus, I needed to draw a line under this one I’d got enough into the dictionaries for 0.11. Readers will remember last time that although I had been able to convert a tremendous number of generators into this more usable and almost infinitely more flexible data structure, which will continue to yield dividends both now and into the future, a ton of them still posed significant challenges. The most substantial of these was that a number of building generators use a technique I’ve sorted of informally termed “offices”. What this means is that a single core grid is selected and then varied to form the basis of the interior layout – that’s fine, all very standard, and precisely the material that I’ve been adding into these dictionaries and taking out of the specific functions that generate the buildings in question. However, in the case of these buildings, some part of the core grid selected are like markers, saying “do something else here later”. During the initial generation the software keeps track of those markers, and then once the foundations of the interior of that building have been laid down, it then looks at that list and knows what to do with them. In most cases it involves spawning blocks of tables and chairs, though in other cases it might be a distribution of money or ingots in vaults (for a bank or a mint), or things of that sort – little blocks of terrain that might appear several times in a generator, but can themselves be varied to add even more variety to the final generated outcome (so if a parliamentary building has twelve “offices”, the layout of each office will be unique, regardless of the overall layout of the interior of the building). This, however, posed a tremendous problem to local map clue generation for various reasons, but primarily about generation not all taking place at once, and the difficulty in trying to pre-select those generative elements into a certain pattern, then integrate them into a normal map grid (i.e. while creating a local map clue), and then trying to ensure the selected parts spawn correctly.
I’m not sure I’m conveying well the trickiness of this – although it could of course be solved by just generating the entire building, but that has game speed implications, so that’s not acceptable – but I really couldn’t find a nice solution here beyond just a tremendous amount of extra, and extremely complex, work. So, the first thing I did this week was that I decided I wanted to get as many of the buildings with the “office” issue into the database as possible, even if it meant extremely minor simplifications to their generative systems. On consideration, I realised that one would need to see many dozens of some of these buildings on the inside – and of course be paying extremely active attention to everything while one does – to even begin to notice this minuscule extra generative variation via the office technique. As such, I decided to just get rid of that, losing perhaps 2% of the generative variation of these buildings, but instantly making them viable for the local map clue generator (and other clue generators in the future which will need to be able to perfectly read and predict the interiors of buildings, or the exteriors of towns or cities, which don’t yet exist at the time of the clue’s creation). This took quite a while to actually put into place for many of these building generators, as these additional components are so closely interwoven into how the generators work, but after a few hours of work across multiple days I’d been able to take quite a few of those I wasn’t able to tackle last fortnight and have them correctly put in place. After this, I turned my attention to some of the major generators that I hadn’t been able to convert yet, such as religious buildings and castles. Both of these have hugely complex generators, are extremely cool and varied interiors, and are big and interesting places for the player to visit, so we obviously want these to be locations that the player is going to be spending some time in. The castle generator, however, remains extremely intimidating in its complexity. I have a few ideas about how to transform it into a format that the local map clue generator can understand, but they’re complicated, and not a priority for 0.11, so I’m going to leave those. Religious buildings, however, didn’t need much changing to get into the generator… but there was one thing. Specifically, as noted last fortnight, I now need to make sure all buildings generate abyss outside them. For example, a grid where “.” corresponds to abyss might look something like this:
That’s all well and good, but the problem is that the religious buildings consist of several hundred grids… and every single one of them needed to be reworked in order to ensure nothing but abyss generated outside of the main play area the player character will actually be walking around within. The issue is that many of the grids, at present, had stuff “left over” from when I was drawing them the first time, e.g. walls that I later decided I didn’t need I didn’t delete, I just left in there and moved on, because the player would never be in a scenario to actually see them (if they were outside the play area). This is obviously a known thing in game development where it can be simpler and easier to leave older things in rather than removing them altogether, and it’s of course on this basis that lots of “cut content” videos on YouTube come into being (I’m a big fan of those Bloodborne ones, personally). As discussed last time, it was very, very hard to think about how to create a system to do this automatically, given just how many strange hidden elements in these generation grids there might be, and the fact that each grid has its own language for what characters mean what, and thus what characters should be kept or lost and which mean what… and after a lot of consideration, it was clear that I needed to do this by hand. I confess upon coming to religious buildings I did go back to this and give it more thought, but the same conclusion still presented itself, and so I went through something like five hundred grids, by hand, to completely transform them. This was… a hell of a task. Some parts of it could be easily automated through the use of Ctrl+C and Ctrl+V and the like, but many parts could not be. But, in making sure that the outsides of buildings generate as abyss rather than outdated or unused physical elements means that these can all be used in the generation of local map clues without the risk of confusing the player, or the risk of breaking the sense of immersion if the player looks at a clue and “sees” that there’s a bunch of stuff floating outside the play area which, ordinarily, could only be visible with some kind of noclip functionality. This took me several days, but was one of those tasks which isn’t cognitively demanding, so I found myself sometimes relaxing into a really good rhythm here – and now, I’m pleased to say religious buildings are sorted!
At the end of this, and with the final things for 0.11 put into these dictionaries – keeping in mind that a few of the building and outdoor generators I am going to leave for 0.12 and beyond, as I just want enough here to ensure massive variety without having to rewrite and rework the most complex generators in the game – I found myself wondering just how many generation grids and other resources were in the overall dictionary. I estimated it would be somewhere around five hundred; there’s around fifty different building types, all with lots of permutations for floors and such, and quite a few elements of outdoor generators were also included in this ongoing transformation process. Perhaps 600? 700? When I asked Python to count, however, I discovered that the generator grid dictionary now contains over 2000 grids, every single one hand-typed / hand-drawn over the last fifteen years, and now all abstracted into a brand new data structure that will enable me to do anything with them in the future, even before they actually generate. I can’t believe the size of the thing, to be honest, but the counting function does not lie. That, I suppose, is what happens when you actually end up spending the better part of fifteen years generating a world that reaches a somewhat unhinged level of variation and complexity (obviously not all the fifteen years have been spent on this exact thing, but you get my point). In turn, I have to say I was actually quite pleased to see the size of this, as it just gives further emphasis to the game’s potential for developing this unique riddle-roguelike gameplay – it’s only when the generative options are so ludicrously vast that the game will be able to guarantee its ability to always produce riddles the player won’t have seen before, no matter how long they’ve been playing for. That’s a really core gameplay objective here, so in that regard, the size of the dictionary is very reassuring. It’s strange, though – I really didn’t feel like I polished over two thousand grids of information as part of this task, but… I guess I did. The thing doesn’t lie. Anyway, with that done, it was then time to shift it over from my testing file and into the game. This dictionary is 60,000 lines long by itself (with 2,000 entries, we perhaps should not be surprised), and then it was time to start testing that everywhere can still actually generate correctly when using this external dictionary, rather than information fully internalto the function that creates the building. I first tried the great hall in the university where the player starts, and wrote a quick function to create an image of the building to check it looked correct. Happily, I got this immediately:
Which is exactly how it should look! Then I quickly enabled a coloured version of the same function just to make sure everything was also generating with the right colours (no reason it shouldn’t, but, this is a huge shift to the game’s data structures and I wanted to be meticulous here), spun up another great hall, and again downloaded its total layout, getting this instead:
Perfect! With this done, I then set about testing out every single interior and exterior area I could find, in which the generative elements had been shifted from the functions themselves into the larger dictionary. Here’s another, with abyss outside though also one tiny bug that I fixed immediately after taking this screenshot (can you spot it?):
And just one more (upper floor, ground floor, basement), noting here how the excess space – since the actual thing is not a perfect square – is also correctly abyssal:
You will note that in the upper floor and lower floor there is quite a bit of spare space. That is another thing but not something to fix in 0.11 – generally for buildings each floor is generated at the maximum possible required size, and is generally generated as a square, even if the building doesn’t need a square to fit itself in and would instead fit perfectly well into a rectangle. All of this is holdover from very early code, and I do indeed to polish it in the future in order to yield further speed increases – though this is not a priority since the speed increases in question for generating buildings would be very minimal, and we actually want to leave space for a future generator that can spawn secrets into some of these floors (i.e. I have a generator planned which will be able to take any location and add a secret room onto it organically (including on the outside, so then a very interesting little observational bit of gameplay will occur when you’re walking past a building – does the exterior match the interior, and if it doesn’t, then there must be a secret room!). This will be incredibly cool, but also complex, and we aren’t doing that just yet). Anyway – at time of writing I’ve been able to get through almost all the generators and confirm they still work, and they do indeed, though in quite a few cases some extra code or bug-fixing was needed for a whole range of different reasons. Sometimes that was also just from my own inability to type with a keyboard like a sensible human, as I discovered a great deal of typos here which I hadn’t been aware of while in the process of shifting things from the in-function generators to the in-dictionary generators. Whoops. Still, as I write this there’s only a handful of the old generators which aren’t yet working correctly with the new material (which is, of course, the old material, but just moved around a bit in the game’s data structure), the most prominent of which is everything to do with towns, which appear to have, er, imploded for some reason. I am honestly not quite sure what has caused this, and I shall be working on fixing this in the coming days and weeks, but right now I can’t quite identify where the issue is coming in. Rest assured, though, that I shall indeed uncover the root cause of this strange and baffling mystery. Nevertheless, the key point is that the integration has gone extremely well (aside from in towns) and I can’t really find places where the shift to this new generation model is causing any major issues whatsoever. This is a great milestone! Still a few left to handle, which I will be handling over the coming weeks, but the major data structure transition has gone fantastically.
Losing the bad walls
If you take a look back at that generated building which I said has an error – can you see it? On the right-hand side there is a wall behind the door, i.e. in what would actually just be air / outside landscape. Given that the player cannot ordinarily see this, I actually used this technique early on in building generation, many years ago, as a method for allowing the game to immediately identify what side of the door the player should be placed on when they entered a building. The game would look at all four sides of the door, take note of which one didn’t contain a wall (and this of course works regardless of orientations, inversions, rotations, etc), and then place the player there; and in turn, it would check where the “middle” wall was of the three that existed (i.e. here north, east, and south exist, so the middle is east) and use that to work out which way the player should be facing when they enter, i.e. the opposite of the middle, so in this case, west. This, like so much else, was totally fine when it was hidden off-screen and the player was never going to see it… but we can’t have that appearing in a local map clue! A wall, behind a door, where actually there is no wall? Absolutely ridiculous, unacceptable. So, this time I was able to create a fast automatic way to do this, happily, although the actual changes still required human input. I wrote up a quick script to search through all 2000+ entries in the generator dictionary, and in each case just look for a “D” or “d” – pretty much the only characters I use for doors – which is surrounded by three “#”s rather than just two (this character’s use for walls is the only universal across every single generator). It then printed me off a list of all the generator grids which had this, and it was rather a lot… but it was, of course, including outside generators too, where you should have a door nestled between three wall tiles. After a little more work I got this script to ignore those, and then just produce a list of grids that were indoors grids but had this remaining visual issue. It took around an hour to go through them all and remove this from all of them, and it certainly wasn’t the most exciting hour of my life, but like so much else in this data restructuring task, it needed to be done, so I got on it. In the process I also found a few remaining issues with how some of the abyss tiles were being handled, too, so I was able to fix that at the same time. With this done, I am now… fairly… confident that everything in this 2000+ entry dictionary is actually looking exactly how it needs to in order to produce reliable local map information that the game can use to generate all manner of cleverness before the player ever even gets there.
The Experiment(s)
This then, finally, took us to the point of doing the First Great Experiment in which we pre-select a particular generation for something, and then generate a local map clue for that generated thing before the generated thing is even generated! Thereby giving the impression that the entire world has always been there, and allowing us to do all kinds of cool things without adding even a second to the world generation. I know I keep saying this, but I do know URR is a little slow (but faster than it has ever been right now, and getting faster with each passing), and we really can’t allow anything to make world generation into an even longer process than it already is. As such, this is the holy grail for getting local map clues working, AND also in the future for getting many other types of clue working as well, all contingent on being able to let the game talk intelligently about an in-game area that doesn’t exist yet, and then guarantee it exists in exactly the right way later on to correctly match the visuals or words used to describe it. So, I created a nice new function – make_local_map_clue – as the testing grounds. I then attached it to the player’s crosshair on the world map for the sake of playtesting, so that you can just select a map tile, have the game create a local map clue for that map tile before generating that map tile, and then you can visit that local map tile and check whether the clue and the map do indeed match up (as they should). Although I said the library on the university campus you start the game on was going to be the initial trial, I actually realised that was a bit awkward precisely because it was where you start the game – you come into being after the map has already generated, and so creating a local map clue before generation, and then giving it to the player afterwards, was going to be just a little bit awkward because of how the code is set up. Instead it would be far easier to have the player spawn in with a clue for literally any other building in any other map tile in the entire world, so I decided to go with that instead. So, instead I again took a look at my list – though a new list this time – of all those which had been successfully moved over into the new generation model, and tried to find the easiest one(s) to start with. Those I marked off with red were all for universities, so I didn’t want to go with those; marked off in green meant they were outside, and outside has complexities I haven’t yet resolved; light blue meant they went into the city centre, and that has its own complexities in how layouts are generated which I need to reverse-engineer; yellow was in military districts, and those also have complexities in how they spawn; and purple were just such damned complex interiors that I really wasn’t sure how to start there. This then left us with four fairly “pedestrian” buildings for the first test – a granary, a farmhouse, an icehouse, or the interior of a warehouse.
However, I quickly moved the farmhouse off the list because many farmhouses can appear on one map grid, which adds a level of complexity, but only one of the other three can ever appear on map grid, making them, finally, the perfect first thing to test. I decided to try with an almshouse first – it could have been any of them, but the almshouse has the most detail on the inside, so let’s go with that one. So, I created a new list of things which the game will always track for which generations have been pre-selected, and called that dictionary forced_generation, in which the keys are map coordinates such as (65,109) and the data lists all pre-selected generative options, for that map grid, if there are any (and if not, then the key just doesn’t exist – otherwise it would automatically be storing 62,500 keys, and that seems like a dreadful waste of time and data). Then I decided on the structure for the tuples that would be used as the information here, and decided that for an outside-only force, it would be something like “(‘Almshouse’, 2, 0, ‘Outside Only’), where the first item is the building or generative thing which is being forced into a certain way when it actually gets generated, the second would be the permutation (Almshouses have four outside variants), the third would be the orientation (eight possibilities on a square grid, i.e. four rotations and two possible reflections for each rotation), and then the fourth part tells us whether there’s anything more in the tuple, or not. If the fourth part is instead ‘More Inside’, then the next item in the tuple would be the inside variation (0, 1, 2, whatever), and then any more information would be the final part if more is needed to ensure unity between the pre-generated version and the final thing being created (that’ll be unique to each generator, and a bit complex, but I am equal to the task). So, I had the game add such a tuple into the forced_generation dictionary for a particular map grid, and then generated a local map clue on that basis. It showed me an area inside an almshouse, with a particular brick colour, in a particular orientation, with a particular layout of furniture, all for an almshouse that did not yet exist in the game in any way, shape or form. Here’s what the clue looked like – obviously it would have an “X” marking something important in the final version! – and with this, I then delved into this nearby city, which I had not yet explored, and had thus so far generated nothing at all within.
Once in, I went to the right map grid and found the almshouse, and then got myself to it. It had generated in this particular layout, and this particular orientation, though without going inside it’s hard to know whether it was the right one or not.
So, I went inside…
…and sure enough, to my absolute delight, THE LOCATION EXISTED, even though the clue was constructed BEFORE the building had ever come into being! I found the right place on the right-hand side (so note the correct orientation, along with everything else) of the building, which looked like this:
And to make the point, here’s the comparison of the two, with the clue having been created before the building was:
So:
It worked!
It worked!!!
I don’t think I need to spell out the “power” this facility now gives. Once everything is done here – this being just one proof-of-concept example – it’ll mean anything can be generated for areas the game hasn’t even generated yet. I’ll be able to have the game create all manner of information for places that don’t exist, up to and including to the level of precise and exact tiles, use that information to create clues or riddles or whatever it might be, and then have those appear correctly when the player finally arrives there… and all without a second of extra world generation time. This is huge! It’s extraordinarily cognitively demanding, I don’t mind saying, but progress is being made at a good, steady rate, and that’s the important thing. With this first test, I know it can be done, and for all the others, it’s just a matter of time now. The system works, it can actually be programmed, and now it’s just going through and adding in all the others. That’ll be a reasonably time-consuming process, but – it has begun. Not just can the game of course use existing information about distant and ungenerated lands to make clues, such as the gods they worship, for example, or things of this sort, but we can now actually have clues generate and force the solutions to come into being when the player finally gets there. This is, in a sense, a really fascinating and almost “backwards” way to do it, and what I now realise is that URR’s quests and riddle generation is going to basically wind up doing a mix of both options. Plenty of stuff is generated at world gen and that stuff is of course going to be used in lots of quests and clues. Equally, as I noted last fortnight, I’ve set the system up so that once you’ve created an entire map grid, the game can use that map grid to create one-off micro-clues for micro-riddles, with small rewards, but very rapid turn-around times (this isn’t implemented yet, but I think this is a really important element for ensuring the entire game world feels as dense and full with mystery as possible). Then, finally, we can have the game create clues for places or scenarios that don’t exist yet, and then when you get there, the correct scenario will exist (looking as if it has always been there), even if we will know (don’t tell anyone…) that it happened the other way around. Through these techniques (world gen -> riddle, local gen -> riddle, riddle -> local gen) we should, once I’ve continued to flesh out and develop the data structures even further, have some absolutely extraordinary possibilities for creating all kinds of amazing stuff. I’m so excited.
What next?
So there we go – we have generated the first ever clue dependent on correctly describing a place that does not exist yet and then ensuring that when that place does exist, it is exactly as depicted in that clue. Whew. This is, in my view, one of the most challenging things I’ll need to do in URR, and it’s so cool to see it working. Only for one building, only in one context – but the proof of concept works. From here it’s just a matter of going through all the others and making sure those all work, and then all the clue types I want in 0.11 will be in, and then it’s quest generation time. I really hope these entries give the sense of the speed of development at the moment, since at this end, I feel as if I’m absolutely blasting along. I wanted one clue type in 0.11, now we have two essentially finished (rhymes and world map) and one coming along incredibly well (local map), and those things are going to provide just so much more variety than the rhyming clues would have been able to do on their own – cool and novel though they certainly are. Really pleased with all this progress. Meanwhile, I’m also pleased to say the soundtrack is coming along really nicely (I know I haven’t talked about this in a while, but it’s still happening!) and I’m continuing, slowly but surely, to solve bug after bug in the background while working on the bigger and more exciting things. As ever, thank you for reading friends – please do leave a comment below with your thoughts, please consider sharing this around on the web with people or communities who might find it Pretty Neat, and with all this done, I’ll see you in a fortnight for more stuff. Hope you have a great couple of weeks!
(Late edit: fixed the town things! Towns now generate in all their beauty once again.)