VS Code comes with a set of standard semantic token types and modifiers for all semantic token providers to use. Still, semantic token providers are free to define new types and modifiers and create a subtype of the standard types.
The standard types and modifiers cover common concepts used by many languages. While each language might use a different terminology for some types and modifiers, by adhering to the standard classifications, it will be possible for theme authors to define theming rules that work across languages.
Along with the standard types and modifiers, VS Code defines a mapping of types and modifiers to similar TextMate scopes. That's covered in the section Semantic Token Scope Map. If necessary, extensions can declare new types and modifiers or create sub types of existing types through the semanticTokenTypes and semanticTokenModifiers contribution points in their extension's package.
In the example above, an extension declares a new type templateType and a new modifier native. By naming type as the super type, the new type will inherit the styling rules that have already been defined for type. Along with custom token types, extensions can define how these are mapped to TextMate scopes.
This is described in the Custom Mappings section. Whether semantic tokens are computed and highlighted is decided by the setting editor. It can have values true , false , and configuredByTheme. Language extensions that depend on semantic tokens can override the default for their language in their package. Theming is about assigning colors and styles to tokens.
Users can also customize the theming rules in the user settings. Two new properties have been added to the Color Theme file format in order to support highlighting based on semantic tokens.
The property semanticHighlighting defines whether the theme is ready for highlighting using semantic tokens. Oct 12, Apr 16, Include ServiceManagement among precompiled headers.
Mar 25, Limit tab size settings for Onigmo submodule to C files. Feb 15, Remove MASPreferences submodule. May 10, Build with Travis.
Jan 15, Remove reference to textmate-dev mailing list. Aug 29, Initial commit. Aug 9, Embryo of developer documentation Those include rules, in turn, are virtually replaced by the rules that their values name — the actual separator rule from the repository, the actual heading rule from the respository, the actual blockquote rule from the repository, and so forth. Those actual rules are all defined in the second-level repository.
Note that the order of the include rules in the patterns array is significant; an array is ordered. The value of an include rule can alternatively be a scope name. This is a good way to inject an entire grammar inside yours. For example, in my AsciiDoc bundle, I assume that the contents of a passthrough block never mind what that is will be XML.
Therefore, inside my match rule for a passthrough block, I have an include rule specifying text. This causes the whole grammar from the XML bundle to come into play in this region of my document. In addition, TextMate 2 permits a single named item to be plucked from a foreign repository using the syntax scopeName itemName.
As I mentioned at the outset, as far as I can tell, if a match rule contains an include key, that is its only key except possibly for comment and disabled. The match key. Its value is a regular expression that the TextMate parser is to look for. This is, I am at pains to stress, a single-line regular expression. This means that any matches we find using the escape rule will be assigned the constant.
A theme or setting may then come along and style any constant. What do I mean by that? Well, the scope name is actually what TextMate calls a format string.
For more about these, see this article. The syntax here is derived from shell programming syntax. With a format string, you can do things such as:. Supply a different result depending on whether a term such as a matched group is defined or not. The captures key. Its value is a dictionary. In this dictionary, each key is a number or name corresponding to a matched group from the match expression, and the corresponding value is a dictionary.
This dictionary may contain one or both of these entries:. A patterns array. This is a list of match rules to be sought within the matched group text — thus permitting the search for matches to continue inside the matched text. With it, the stretch of matched text itself becomes a candidate for further matches. Imagine a toy markup language in which bold is delimited by asterisks and italic is delimited by underlines:.
This works, but only for bold and italic stretches of text that are separate , like this:. An italic stretch cannot occur inside a bold stretch, because once a stretch of text has been matched as bold, it is discarded from further consideration. That sort of thing is perfectly legal — the patterns array is a list of match rules, and we are providing match rules.
And it works, correctly marking up text such as this:. So, for example, this is wrong :. You are getting the benefit of my hard-fought experience here; it took me about a week of wrestling and some direct help from Allan Odgaard to learn the right way to do an include inside a one-pattern match rule.
A two-pattern rule must contain two regular expressions: either begin and end , or new in TextMate 2 begin and while. These are still single-line regular expressions the TextMate parser does not consider more than one line at a time , but there are two of them, so together they may delimit a stretch of text that embraces multiple lines.
If the begin regular expression generates a match, TextMate will start looking immediately after it starting in the same line, if the begin match did not snarf up the entire line for the end or while expression. If you provided an end expression, TextMate will keep walking the document until it matches the end expression, and will stop.
If you provided a while expression, TextMate will keep walking the document until it comes to a line where the while expression fails. This is a major limitation in the degree of intelligence a language grammar can exhibit; it is said to be due to the need for speed, and perhaps for simplicity.
In any case, it is intentional, and some grammars actually take deliberate advantage of this behavior. For example, the Property List Old-Style grammar starts like this:. The expression? This end pattern is designed to fail. The whole pair thus means: Once you have encountered a left parenthesis or a left curly brace, immediately snarf it up along with the entire remainder of the document and just about everything else in the grammar then works through include rules inside that snarfed-up material.
Thus, if the begin is encountered and the while is never encountered, only the rest of the line containing the begin text is subsumed into the matched text.
After encountering a begin match, the TextMate parser looks in the next line to see if it can match the while pattern. If it can, it incorporates the matched text and the entire rest of that line , and looks in the next line to see if it can match the while pattern. This continues until a line is encountered in which the while pattern cannot be matched. Again, there is no backtracking; failure to find the while pattern at all is not a reason for rejecting the begin match that was already found.
Notice also that nothing about these rules says that the resulting matched stretches of text have to be contiguous. Then matching stretches are shown in caps :.
Well, I told you it was strange! For example, the blockquote rule is written like this:. That means, in essence: match an entire line that begins with one begin or more while greater-than signs and perhaps one space after the greater-than sign , and keep matching entire successive lines while that begin that way. The name key. This is the scope or an expression evaluating to a scope to be applied to the entire matched stretch es of text starting at the start of the begin match. The contentName key.
The beginCaptures and endCaptures or whileCaptures keys. These work like the captures key in the one-pattern rule: the value is a dictionary each of whose entries refers to a match group in the begin , end , or while pattern respectively, and is itself a dictionary containing name or patterns or both. If the name or number of the matched group happens to be the same for both the begin pattern and the other pattern, you can if appropriate use captures as a shorthand to avoid saying the same thing twice.
The patterns key. This is like the patterns key inside the captures and beginCaptures and so forth dictionaries, but it applies to the region between the begin and end matches. Again, this is so that you can continue searching inside this stretch of text, which otherwise would be marked down as completed and TextMate would start matching after the end match.
A surprise arises if a match specified in the patterns array can be satisfied by continuing beyond the end match. What happens is that the match does continue beyond the end , and causes the end to shift with it. For example, suppose we have this rule:.
I find this both surprising and disappointing, as it greatly limits the kinds of natural logical structure you can successfully express. In addition, we are told that, in case both a subpattern from the patterns array and the main end pattern would end exactly on the same character, the end pattern wins unless you set the applyEndPatternLast key to 1. Now create your initial language grammar and start writing rules.
As you create rules, add test text to the document so that you can see how it is affected. The big issue, of course, is whether a given stretch of text is being assigned the scope you think it should be. Unfortunately, TextMate gives you no simple way to discover this.
0コメント