Although the concept and design of the Data Modules section for StoryDev seemed like a good idea, it has since hit a dead-end. Progress has been slow which is normally an indicator in the back of the mind that it is not working out as expected.

Normally, when dead-ends such as this occur, it is best to return to the drawing board and wait until a new idea comes to mind.

This will happen often and it may seem spontaneous or unusual, especially when StoryDev is still in Beta and features are yet to be developed.

Introducing ExtraScript

A potential solution to this dead-end is the idea of a scripting language combining data sets with scripting.

The concept of this language is to expressly define data structures without needing to specify data types and store such data in a human-readable way. It would be JSON-like, with added functionality and with performance in mind.

scope data as NewsItem;

blog1 = {
  (ID: 1),
  (Title: The Idea of ExtraScript),
  (Brief: A discussion about a new language in concept.),
  (Description: Maxime et magnam officia earum aut omnis quos in. Laudantium blanditiis dignissimos quia iure tenetur quas ut reiciendis. Beatae vel ea ab repellendus. Tempora id voluptas iure porro in tempora animi corrupti. Quia et animi porro neque sit consequatur labore excepturi.
Rem quae et et tenetur. Aut esse quasi totam nisi autem commodi. Nemo aut cumque nam. Delectus et repellendus ullam in reiciendis vel atque odit. Nemo pariatur assumenda fuga atque quos sed repellendus.),
  (Authors: [tienery]),
  (Tags: [General, StoryDev, ExtraScript]),
  (DateUpdated: 31/08/2021)

Not another data format! We already have JSON, XML, YAML, SQL and others on the market, why do we need another one?

Well, unlike any of these formats, the aim is NOT to be a data format. In fact, the idea behind Data Modules in StoryDev was to make custom data, but I want to allow you, as the user, the ability to be as flexible as possible with these data formats, in exactly whichever way you choose.

That is why ExtraScript is actually going to be made up of three components:

  • Data Structure
  • Compile (if required)
  • Execute

Data Structure

The first thing ExtraScript will do, where the bulk of the work will be performed, is determine scopes, as you see above, and then interpret the content as per that scope.

The scope data is where most of the parsing and interpreting is done.

The example above is very simple and this practice is likely rarely to be used for the purposes of ExtraScript. However, for the purposes of explaining the concept, it is what we will use.

In this case, ExtraScript is going to go through the entire data structure tree in each object and identify each key, adding it up until it it forms the final Data Structure. This could take some time, and this would be the first pass. This would then be cached to speed up the first pass next time, only checking for changes in the data structure if needed.

This means that you do not have to think about the structure of your data at all. If you need a field for one object but not the other, ExtraScript will default to the inferred type's default value. For example, if an integer, it will default to 0.

Next, ExtraScript will determine, based on specified properties, what and how data should be stored.

By default, I would expect the default behaviour for storage to be a single Array. Each "variable" declared like blog1 would be construed as an integer specifying the index of the data in the global variable stored as NewsItem, hence why we specify as NewsItem as above.

To put it in code context, our final code generated would look like this:

package data;

class ExtraData
    public static var NewsItem_DataSOA:NewsItemSOA;

class NewsItemIndices
    public static inline var blog1:Int = 0;

class NewsItemSOA
    public var ID:Array<Int>;
    public var Title:Array<String>;
    public var Brief:Array<String>;
    public var Description:Array<String>;
    public var Authors:Array<Array<String>>;
    public var Tags:Array<Array<String>>;
    public var DateUpdated:Array<Date>;

This is only an approximate result. There may be changes to this concept when implemented in practice, but this is an example of what we may expect to see when generating the results we need.

There will be more complicated scenarios that will take place, including function handlers, enumerations, associations, templates and more, but for now it may be worth keeping it simple and expanding later.

Compile Scope

In this scope, all we are doing is taking the code beneath scope compile; and interpreting it as Haxe macro code. However, an opportunity arises from this, which is to use ExtraScript as a method to more accurately pinpoint issues at the pre-compiler level, which, at the moment, is difficult to debug in Haxe.

Execute Scope

Or, in other words, scope script;. Anything below this will be interpreted as Haxe runtime code.

Runtime code would be executed based on the file name in which you are coding and in what context.

As such, it would not be unusual to see many parameters against a scope script; to determine when or how the underlying code is executed.

It could be that the underlying code should be generated as a separate class, or perhaps part of an existing function. It would, of course, be wise to generate this code using Haxe macros -- it's the only option.

Of course, this does mean slower compile times, but hopefully we can make the performance of the Data stage as fast as possible so that this portion does not feel so clumsy or slow.

What about StoryDev?

StoryDev will currently be on hold to focus on this new language.

As this new language will effectively become the de facto method of data structuring and storage for StoryDev, any UI previously planned for such development will also be put on hold until a workable version of ExtraScript becomes available.

Considering the data that already exists in our current project, there is enough to go by to test this new language.

Developer at home, customer services at work. In my free-time, I enjoy writing and coding.