Updated: Jul 27, 2021
Hey! How's it going?!
I'm glad to see you're still interested and want to learn more. In this post, we will uncover one of - to me at least - the most interesting aspects of CM: Syntax Extensions.
If you are familiar with languages such as Lisp (Lisp's Macros are especially fun and powerful when compared to the others in this list), Python, C++, and Boo (or to a lesser degree, the .Net platform) you might already have leveraged the preprocessor's capabilities beyond the "standard" language specification into the realm of meta-programming.
The key aspect here is to be able to expand the SLE for whatever platform you are working on into something that fits your requirements in a more efficient and/or friendly manner.
You can potentially use this capability to:
Create a DSL to better map the code to the business jargon of your project;
Add new features to the underlying GPL you are working with;
Create templates to reduce code redundancy;
Without further ado, I think it's time for us to get to work and create our first CM Syntax Extension (which from now on I'll refer to simply as CSE).
There are several different "types" of syntax structures in CM. You can find them all just by searching for “extends Syntax” inside the base CM repository.
This list contains all of Core's standard CSE-related structures. One could argue that even though the topic itself is somewhat dense, at least we don't have a whole lot of different classes to learn, right?
In fact, we are not even going to explore all of these just yet. For the purpose of this post, we are only going to look into a few of those and try to get a really solid understanding of what really is this CSE thing and how this infrastructure works so we can leverage this knowledge into the next steps of our journey towards CM Closures.
For our very first syntax extension, we are going to work with Statements.
...a syntactic unit of an imperative programming language that expresses some action to be carried out... A statement may have internal components (e.g., expressions). - Wikipedia
I think that based on the above description extracted from Wikipedia you can tell that I kinda lied a little bit when I said that we'd be looking first into Statements. That's because in order to understand and utilize statements you first need to create the expressions that will be part of it. Worry not though, CSE got you covered!
I don't really think I could do a better job explaining these concepts with my own words so, let's start by taking a look at what Wikipedia also has to say about Expressions:
...a syntactic entity in a programming language that may be evaluated to determine its value... a combination of one or more constants, variables, functions, and operators ... to produce ("to return", in a stateful environment) another value.
In short: Expressions are instructions that produce - in CET's case also return - a value inside your program.
Luckily for us, Core also brings some CSE expression as part of its source code and you can - and should - look what's under the hood by searching "public expr" in your base directory.
We are not going to explore any existing syntax tough. I believe we are capable of starting off with something simple but complete enough to provide you with sufficient understanding to navigate your way across the several examples you will find in Core, all by yourself.
Time 2 Code
Our first Expression syntax will be, as you probably guessed, a Hello World. We will create a syntax that when used will pln and return the message "Hello World from syntax".
Here's the code we will be using:
Go ahead, run this and you will see the following output:
Hey... Hello World from syntax Hello World from syntax Cool! cm>
Looking at the code I gave you do you see what we did there (in terms of basic syntax expansion)?
Let's explore the highlights of our little experiment:
We've expanded the semantic behind the exclamation point and made it part (a required part, by the way) of our newly created expression;
We've successfully wrapped a set of instructions inside a construct that CM preprocessor will evaluate during compile time;
Under the Hood
Let's expand on the previous highlighted items and add some more information on how this magic works behind the scenes while introducing some points you should keep in mind while working with CSE.
The easiest way to explain how this works is if you picture the usage of CSE's as if it were using a small Bot that goes around your CM code finding all the places from which you called it and replacing the code in there with the code you've placed inside its own definition file.
Our little Bot friend is smart enough to preserve the context in which you have called it while also being versatile allowing parameterization in a way that it can replace the placeholder call with the exact expressions, statements, and/or members (we will cover those in the future) you've asked it to put in there.
To better illustrate this Bot abstraction let's take a look at what actually is compiled when we run the helloWorld! extension above:
That's right, no sign of the original syntax declaration. It's gone. Our Bot friend is not part of the final code, it is only used to generate the code that will be used as the final code.
CSE Source Code Update
Understanding that the syntax code doesn't actually exist anywhere on the final code is an important part of a happy life while coding for it. It could get really frustrating very fast if you are trying to do something using Syntax and the compiler is yelling at you or, as it happens more often than it should, your code just won't update at all.
Say you have the following files:
For simplicity's sake, let's assume that both these files are located under the same directory and share the same package.cm file and package references.
Here is the source code for both files:
All pretty standard at this point, right? It is, and in fact, it will actually work exactly as you would expect based on what we've seen so far.
Go ahead and compile those files and run the code in myClass.cm. You will see something like this:
Hello World from syntax Hello World from syntax Hi! cm>
So what's the issue here?
Well, nothing yet. We will get there!
Still using these two classes, say I change the message in my syntax file to something else. Here's the updated code:
Now go ahead and save+run this file. It will compile just right and you would think that you are all set.
But, as soon as you go to your unchanged MyClass.cm file and run it (you can even re-compile it if you want) you will notice that the message displayed did not change!
Is this real life?
As we previously established, just like the spoon in Matrix, there's no Syntax.
The source code for the syntax is only used by the preprocessor/compiler while validating/generating CM's machine code and, after that, syntax declarations are considered "safe to ignore" since, in theory, its purpose was fulfilled, right?!
What if, like in our example, you change an existing syntax declaration? How does the preprocessor/compiler know that it needs to go ahead and redo the work on top of the files where that syntax is used?
The short answer is: it doesn't.
In order for you to see the changes, you've made on a Syntax definition replicated into the files where such syntax is used you need to "force" the compiler to run on top of it again. Actually, not only that. you also need to convince the compiler that it needs to run on that file.
I might be making it sound worse than it really is here, so here are the two options you have:
If you have a single file that you want to see those changes applied to (either because that's the only place where you use it or this is a test file) all you need to do is add an empty space anywhere in it. That will let the compiler know that the previous definition of that file was likely changed, therefore, a new build needs to be executed against it.
If you have several files where that syntax is used and you want them all to be updated at the same time, then the only option you have (that I know of) is to clean and recompile your entire workspace.
If you want to take it for a spin yourself, go ahead and change the MyClass.cm file and run it again. You will see this as the new output:
Over thinking, Over analyzing, separate the body from the mind Over thinking, Over analyzing, separate the body from the mind Hi! cm>
If you made it this far, congratulations! You now know how to create syntax using CM and I'm proud of it!
Next up we will look into CM Syntax Statements and how they differ from Expressions.
Feel free to drop me a line if you have any questions on any of the topics in here and I'm always open to hearing what you think about the content in here.
Stay safe and see you next time!