lex: rm lex::Kind::Whitespace

Whitespace tokens do not necessarily create new events but they work as
a delimiter for words with attributes and affect some container
delimiters. Now when we can read the source from inline, we can instead
inspect for whitespace when needed.

Removing the whitespace token allows the lexer to continue a lot longer
without stopping. E.g. a typical line in a paragraph with no special
characters can turn into a single token.
This commit is contained in:
Noah Hellman 2023-02-19 18:28:00 +01:00
parent 9454a2e393
commit 86ee4ee520
3 changed files with 40 additions and 43 deletions

View file

@ -910,9 +910,7 @@ impl<'s> Parser<'s> {
inline::Atom::Escape => Event::Escape,
},
inline::EventKind::Str => Event::Str(inline.span.of(self.src).into()),
inline::EventKind::Whitespace
| inline::EventKind::Attributes { .. }
| inline::EventKind::Placeholder => {
inline::EventKind::Attributes { .. } | inline::EventKind::Placeholder => {
panic!("{:?}", inline)
}
}