Vento meets Sublime Text

Recently I've become increasingly interested in Vento, and became motivated to contribute to help improve the language and broaden its audience.

The primary reason I had not looked into Vento earlier than I did, given the passel of Eleventy users flocking towards it, is because it didn't have a Sublime Text integration at all. Really, all I want is syntax highlighting; in the past, I had written my own templating language, CSML (probably don't use it, use Vento instead) and used it for a while to build the website for Yozo. Ultimately, while I liked the language enough, the lack of any form of syntax highlighting got to me, and I wasn't motivated enough to get through the process of creating a Sublime syntax definition for it (especially because it was an indentation-based language), do dropped it and migrated to Eleventy.

Vento already has a more established user base, so I feel much less disinclined to put some effort into creating a better Sublime experience for it. So, before starting my journey of actually writing Vento, I decided to create a basic syntax definition for Sublime.

Vento is mostly JavaScript

Luckily for me, and this is part of the reason I connect well with Vento, is that it tries to utilize JavaScript as much as possible. It avoids reinventing the wheel, and this means a syntax definition can yank most things from the existing built-in JavaScript definition.

But, naturally, it's not that easy. Vento uses double curlies to determine which part is JavaScript and which part is template content; for example, when writing {{ "}}" }}, it knows that the first {{ open a "tag", containing JavaScript, and it also knows that the }} within the string is still part of the JavaScript, and is not closing the {{ it saw earlier. The second pair of curlies it knows that it's in the top-level JavaScript context and therefore these curly brackets are closing the tag.

In Sublime syntax definitions, there is the concept of "embedding a language" within another (see Sublime's docs on Syntax Definitions), but this system relies on greedily breaking out of the embedded language as soon as possible. This is generally fine, for example, when highlighting the contents of <script> tags in HTML, the embedding must end as soon as a </script> is encountered, regardless of where it is found. Similarly, in Markdown, an embedded snippet must end when the closing triple backticks are found, even if it is in the middle of a string according to the embedded language. Anyway, for Vento, it doesn't work that way; so I needed another way to embed JavaScript, such that it only attempts to highlight Vento-level tokens if it is not already busy consuming JavaScript. And so I tried, but then there were pipes, and hyphens.

Pipes and hyphens

Probably the two most difficult-to-solve problems were dealing with Vento's pipes, |>, and the hyphen in a closing tag marker, -}}, indicating whitespace trimming.

In Vento, the pipes are placed between JavaScript expressions in some contexts, such as {{ foo |> toUpperCase() }}; it's effectively like filters in Liquid or Nunjucks, but inspired by the JavaScript proposal for the pipeline operator. The issue is that that is only a proposal, and so Sublime's native JavaScript highlighting doesn't recognize |> as a single operator but rather as a bitwise operator followed by a greater-than operator. At first, I thought this was alright, because it highlights as an operator regardless, but as I implemented more cases, I realized it wasn't so pretty because some contexts require the definition to specifically match the pipeline operator. In particular, when writing {{ echo |> foo }}, the content following this tag is auto-escaped (not interpreted as Vento, even if it looks the part) until a closing {{ /echo }} is encountered. On the other hand, {{ echo "foo" }} renders only foo and does not expect to be closed like the first example does. That is to say; the pipe is a hard switch between two different modes, and they must be highlighted with different rules. While writing this, I realize I probably didn't have to match the |> within {{ echo |> foo }} in order to highlight it properly, I just needed a lookahead to distinguish the cases and then highlighting the whole |> foo }} as JavaScript would work okay. But that's in retrospect, and I'm kind of glad I didn't do that.

Hyphens pose a different, more pressing problem, because applying the JavaScript syntax definition naively until it "errors" back to the top-level context simply doesn't highlight correctly. An example case that fails this is {{ 23 -}}, because the 23 - is valid JavaScript, at least up to that point, and then the unexpected } causes it to "error" out of the subtraction context back to the top level. The }} then is highlighted correctly as the tag's closing punctuation. The result is that the - is highlighted as an operator, whereas it should be highlighted as part of the -}}, a different type of punctuation. This is not something I could see myself bypassing with a reasonably simple hack, like I could with the pipeline operator, so instead I chose a route that would work decently for both these issues.

The solution I ended up going for is to create two seperate, hidden languages:

• One for "JS expressions in Vento", which inherits from the JavaScript language and adjusts the definition to "stop" an expression when it sees a |>, a -}} or a }} coming (without matching it).
• The other for "JS in Vento", used by the {{> … }} tag; this tag allows whole blocks of JavaScript, including flow control and the like, not only expressions. This tag does not allow pipes, and so doesn't have specific rules breaking out of its context when it is about to consume a |>, but only reacts to -}} and }}.

I don't quite like the fact that I made two additional syntax definition files only to achieve this, and perhaps Sublime's with_prototype could be of use here, but I honestly can't really be bothered to understand what a "prototype" means in terms of Sublime syntax definitions. It's not so bad, anyway, to split the JavaScript-specific bits off into their own files.

Testing

This was a part I wasn't necessarily looking forward to, but actually ended up being quite enjoyable and prompted a good few adjustments to the initial version of the syntax definition.

I had seen syntax definition tests before, so I understood how they were written, but I had no clue how to run them. Trying to figure that out, I stumbled upon a UnitTesting package, but ultimately didn't understand how that hooked into the type of tests I'd seen in the wild. Turns out, running syntax definition tests is just built into Sublime text, and all I had to do is symlink the package into my Packages directory so that Sublime can see it and then running the "Build" command (ctrl + B) while looking at the test file.

    {{ if foo == 'bar' }}
{{# ^^ punctuation.definition.tag.start.vento #}}
{{#    ^^ keyword.control.conditional.if.vento #}}
{{#       ^^^^^^^^^^^^ source.js.embedded.vento #}}
{{#                    ^^ punctuation.definition.tag.end.vento #}}

I found it to be surprisingly satisfying to write these types of tests, because I got to write them in Vento (as in, the test itself is highlighted as Vento) and all the assertions work perfectly fine inside Vento-style comments. So effectively, the file becomes a series of repeated blobs that start with a line of highlighted Vento, followed by a small heap of commented-out assertions about what type of scope is assigned to what character from the leading line of Vento. One thing that really did help a lot is that Vento's syntax is not particularly complicated; the syntax definition is not long, and so the test files weren't particularly long either. Walking through each different tag was nice, making good progress with each assertion, being able to tick them off consistently without too much effort.

Making it a plugin

So far, everything's been running locally, and that's enough for me to work with it, but obviously is not the end goal. I don't quite understand (yet) how Sublime's package system really fits together, because it seems packagecontrol.io is more of a third-party thing even though it is well-integrated into Sublime Text. To submit a new plugin, apparently one must fork another repository (not even under Sublime's scope, but presumably the owner is a Sublime employee) and create a pull request to add your package to a JSON file. I admit I was hoping for a more mature workflow there, but alas, this is what we have to work with. Either way, that's for another day.