Please see the changelog for details!
const Lexer = require('snapdragon-lexer');
const lexer = new Lexer();
lexer.capture('slash', /^\//);
lexer.capture('text', /^\w+/);
lexer.capture('star', /^\*/);
console.log(lexer.tokenize('foo/*'));{%= apidocs("index.js") %}
Register a handler function.
Params
type{String}fn{Function}: The handler function to register.
Example
lexer.set('star', function(token) {
// do parser, lexer, or compiler stuff
});As an alternative to .set, the .capture method will automatically register a handler when a function is passed as the last argument.
Get a registered handler function.
Params
type{String}fn{Function}: The handler function to register.
Example
lexer.set('star', function() {
// do parser, lexer, or compiler stuff
});
const star = handlers.get('star');Type: {boolean}
Default: true (contant)
This property is defined as a convenience, to make it easy for plugins to check for an instance of Lexer.
Type: {string}
Default: ''
The unmodified source string provided by the user.
Type: {string}
Default: ''
The source string minus the part of the string that has already been consumed.
Type: {string}
Default: ''
The part of the source string that has been consumed.
Type: {array}
Default: [] (instance of [snapdragon-stack][])
Array of lexed tokens.
Type: {array}
Default: [''] (instance of [snapdragon-stack][])
Array of captured strings. Similar to the lexer.tokens array, but stores strings instead of token objects.
Type: {array}
Default: [] (instance of [snapdragon-stack][])
LIFO (last in, first out) array. A token is pushed onto the stack when an "opening" character or character sequence needs to be tracked. When the (matching) "closing" character or character sequence is encountered, the (opening) token is popped off of the stack.
The stack is not used by any lexer methods, it's reserved for the user. Stacks are necessary for creating Abstract Syntax Trees (ASTs), but if you require this functionality it would be better to use a parser such as [snapdragon-parser][snapdragon-parser], with methods and other conveniences for creating an AST.
Type: {array}
Default: []
FIFO (first in, first out) array, for temporarily storing tokens that are created when .lookahead() is called (or a method that calls .lookhead(), such as .peek()).
Tokens are dequeued when .next() is called.
Type: {Object}
Default: { index: 0, column: 0, line: 1 }
The updated source string location with the following properties.
index- 0-indexcolumn- 0-indexline- 1-index
The following plugins are available for automatically updating tokens with the location:
- [snapdragon-location][]
- [snapdragon-position][]
Type: {string}
Default: undefined
The source of the input string. This is typically a filename or file path, but can also be 'string' if a string or buffer is provided directly.
If lexer.input is undefined, and options.source is a string, the lexer will attempt to set lexer.input by calling fs.readFileSync() on the value provided on options.source.
Type: {string}
Default: undefined
If options.mode is character, instead of calling handlers (which match using regex) the .advance() method will consume and return one character at a time.
Type: {string}
Default: undefined
Specify the token property to use when the .push method pushes a value onto lexer.stash. The logic works something like this:
lexer.append(token[lexer.options.value || 'value']);See the [snapdragon-token][] documentation for more details.
Plugins are registered with the lexer.use() method and use the following conventions.
Plugins are functions that take an instance of snapdragon-lexer.
However, it's recommended that you always wrap your plugin function in another function that takes an options object. This allow users to pass options when using the plugin. Even if your plugin doesn't take options, it's a best practice for users to always be able to use the same signature.
Example
function plugin(options) {
return function(lexer) {
// do stuff
};
}
lexer.use(plugin());