Hey! How's it going?!
I'm glad you're still interested and want to learn more. In this post, we will uncover one of - to me at least - the most interesting aspects of CM: Syntax Extensions.
If you are familiar with languages such as Lisp (Lisp's Macros are especially fun and powerful when compared to the others in this list), Python, C++, and Boo (or, to a lesser degree, the .Net platform), you might already have leveraged the preprocessor's capabilities beyond the "standard" language specification into the realm of meta-programming.
The key aspect here is to expand the SLE for whatever platform you are working on into something that fits your requirements more efficiently and/or is friendly.
You can potentially use this capability to:
Hello Syntax
Without further ado, I think it's time for us to get to work and create our first CM Syntax Extension (which, from now on, I'll refer to simply as CSE).
There are several different "types" of syntax structures in CM. You can find them all by searching for “extends Syntax” inside the base CM repository.
This list contains all of Core's standard CSE-related structures. One could argue that even though the topic is somewhat dense, at least we don't have many different classes to learn, right?
We are not even going to explore all of these just yet. For this post, we are only going to look into a few of those and try to get a really solid understanding of what this CSE thing is and how this infrastructure works so we can leverage this knowledge into the next steps of our journey toward CM Closures.
For our very first syntax extension, we will work with Statements.
...a syntactic unit of an imperative programming language that expresses some action to be carried out... A statement may have internal components (e.g., expressions). - Wikipedia
I think that based on the above description extracted from Wikipedia, you can tell that I kinda lied a little bit when I said that we'd be looking first into Statements. That's because in order to understand and utilize statements, you first need to create the expressions that will be part of it. Worry not, though; CSE got you covered!
CSE's Expressions
I don't think I could do a better job explaining these concepts with my own words so let's start by taking a look at what Wikipedia also has to say about Expressions:
...a syntactic entity in a programming language that may be evaluated to determine its value... a combination of one or more constants, variables, functions, and operators ... to produce ("to return", in a stateful environment) another value.
In short: Expressions are instructions that produce - in CET's case, also return - a value inside your program.
PRO-TIP
Luckily for us, Core also brings some CSE expression as part of its source code, and you can - and should - look at what's under the hood by searching "public expr" in your base directory.
We are not going to explore any existing syntax tough. I believe we can start with something simple but complete enough to provide you with sufficient understanding to navigate your way across the several examples you will find in Core, all by yourself.
Time 2 Code
As you probably guessed, our first Expression syntax will be a Hello World. We will create a syntax that, when used, will pln and return the message "Hello World from syntax".
Here's the code we will be using:
Go ahead, and run this, and you will see the following output:
Hey...
Hello World from syntax
Hello World from syntax
Cool!
cm> Looking at the code, I gave you do you see what we did there (in terms of basic syntax expansion)?
Let's explore the highlights of our little experiment:
We've expanded the semantic behind the exclamation point and made it part (a required part, by the way) of our newly created expression;
We've successfully wrapped a set of instructions inside a construct that the CM preprocessor will evaluate during compile time;
Under the Hood
Let's expand on the previously highlighted items and add more information on how this magic works behind the scenes while introducing some points you should keep in mind while working with CSE.
Syntax resolution
The easiest way to explain how this works is if you picture the usage of CSEs as if it were using a small Bot that goes around your CM code, finding all the places from which you called it and replacing the code in there with the code you've placed inside its definition file.
Our little Bot friend is smart enough to preserve the context in which you have called it while also being versatile, allowing parameterization in a way that it can replace the placeholder call with the exact expressions, statements, and/or members (we will cover those in the future) you've asked it to put in there.
To better illustrate this Bot abstraction, let's look at what actually is compiled when we run the helloWorld! extension above:
Wait, what?!
That's right, no sign of the original syntax declaration. It's gone. Our Bot friend is not part of the final code, it is only used to generate the code that will be used as the final code.
CSE Source Code Update
Understanding that the syntax code doesn't exist anywhere on the final code is an important part of a happy life while coding for it. It could get frustrating very fast if you are trying to do something using Syntax and the compiler is yelling at you or, as it happens more often than it should, your code just won't update at all.
Say you have the following files:
helloWorldSyntax.cm
myClass.cm
For simplicity's sake, let's assume these files are located under the same directory and share the same package.cm file and package references.
Here is the source code for both files:
HelloWorldSyntax.cm
myClass.cm
All pretty standard at this point, right? It is, and in fact, it will work exactly as you would expect based on what we've seen so far.
Compile those files and run the code in myClass.cm. You will see something like this:
Hello World from syntax
Hello World from syntax
Hi!
cm> So what's the issue here?
Well, nothing yet. We will get there!
Still using these two classes, say I change the message in my syntax file to something else. Here's the updated code:
Now go ahead and save+run this file. It will compile just right, and you will think you are all set.
But, as soon as you go to your unchanged MyClass.cm file and run it (you can even re-compile it if you want), you will notice that the message displayed did not change!
Is this real life?
As we previously established, just like the spoon in Matrix, there's no Syntax.
The source code for the syntax is only used by the preprocessor/compiler while validating/generating CM's machine code. After that, syntax declarations are considered "safe to ignore" since, in theory, their purpose was fulfilled, right?!
Well, kind of.
What if, like in our example, you change an existing syntax declaration? How does the preprocessor/compiler know that it needs to go ahead and redo the work on top of the files where that syntax is used?
The short answer is: it doesn't.
To see the changes you've made on a Syntax definition replicated into the files where the syntax is used, you need to "force" the compiler to run on top of it again. Actually, not only that. You also need to convince the compiler that it needs to run on that file.
I might be making it sound worse than it is here, so here are the two options you have:
If you have a single file that you want to see those changes applied to (either because that's the only place where you use it or this is a test file), all you need to do is add an empty space anywhere in it. That will let the compiler know that the previous definition of that file was likely changed; therefore, a new build needs to be executed against it.
If you have several files where that syntax is used and want them all to be updated simultaneously, then the only option you have (that I know of) is to clean and recompile your entire workspace.
If you want to take it for a spin, go ahead and change the MyClass.cm file and run it again. You will see this as the new output:
Over thinking, Over analyzing, separate the body from the mind
Over thinking, Over analyzing, separate the body from the mind
Hi!
cm> Final words
Suppose you made it this far; congratulations! I'm proud that you now know how to create syntax using CM!
Next, we will examine CM Syntax Statements and how they differ from Expressions.
Feel free to drop me a line if you have any questions on any of the topics here, and I'm always open to hearing what you think about the content here.
Stay safe, and see you next time!
