lex: rm lex::Kind::Whitespace
Whitespace tokens do not necessarily create new events but they work as a delimiter for words with attributes and affect some container delimiters. Now when we can read the source from inline, we can instead inspect for whitespace when needed. Removing the whitespace token allows the lexer to continue a lot longer without stopping. E.g. a typical line in a paragraph with no special characters can turn into a single token.
This commit is contained in:
parent
9454a2e393
commit
86ee4ee520
3 changed files with 40 additions and 43 deletions
|
@ -910,9 +910,7 @@ impl<'s> Parser<'s> {
|
|||
inline::Atom::Escape => Event::Escape,
|
||||
},
|
||||
inline::EventKind::Str => Event::Str(inline.span.of(self.src).into()),
|
||||
inline::EventKind::Whitespace
|
||||
| inline::EventKind::Attributes { .. }
|
||||
| inline::EventKind::Placeholder => {
|
||||
inline::EventKind::Attributes { .. } | inline::EventKind::Placeholder => {
|
||||
panic!("{:?}", inline)
|
||||
}
|
||||
}
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue