* Move tests into own file
* Move data structs to own file
* Move functions parsing 1 Token (primitives) into own file
* Rename param_flag_list to signature
* Add tests
* Fix clippy lint
* Change imports to new lexer structure
* Document the lexer and lightly improve its names
The bulk of this pull request adds a substantial amount of new inline
documentation for the lexer. Along the way, I made a few minor changes
to the names in the lexer, most of which were internal.
The main change that affects other files is renaming `group` to `block`,
since the function is actually parsing a block (a list of groups).
* Further clean up the lexer
- Consolidate the logic of the various token builders into a single type
- Improve and clean up the event-driven BlockParser
- Clean up comment parsing. Comments now contain their original leading
whitespace as well as trailing whitespace, and know how to move some
leading whitespace back into the body based on how the lexer decides
to dedent the comments. This preserves the original whitespace
information while still making it straight-forward to eliminate leading
whitespace in help comments.
* Update meta.rs
* WIP
* fix clippy
* remove unwraps
* remove unwraps
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
Co-authored-by: Jonathan Turner <jonathan.d.turner@gmail.com>
* Put parse_definition related funcs into own module
* Add failing lexer test
* Implement Parsing of definition signature
This commit applied changes how the signature of a function is parsed. Before
there was a little bit of "quick-and-dirty" string-matching/parsing involved.
Now, a signature is a little bit more properly parsed.
The grammar of a definition signature understood by these parsing-functions is
as follows:
`[ (parameter | flag | <eol>)* ]`
where
parameter is:
`name (<:> type)? (<,> | <eol> | (#Comment <eol>))?`
flag is:
`--name (-shortform)? (<:> type)? (<,> | <eol> | (#Comment <eol>))?`
(Note: After the last item no <,> has to come.)
Note: It is now possible to pass comments to flags and parameters
Example:
[
d:int # The required d parameter
--x (-x):string # The all powerful x flag
--y (-y):int # The accompanying y flag
]
(Sadly there seems to be a bug (Or is this expected behaviour?) in the lexer, because of which `--x(-x)` would
be treated as one baseline token and is therefore not correctly recognized as 2. For
now a space has to be inserted)
During the implementation of the module, 2 question arose:
Should flag/parameter names be allowed to be type names?
Example case:
```shell
def f [ string ] { echo $string }
```
Currently an error is thrown
* Fix clippy lints
* Remove wrong comment
* Add spacing
* Add Cargo.lock
