Free Lexical Analyzer / Scanner Source

Question

I'm looking for a hand-written free lexer/scanner for any language, even a toy language written in a language similiar to C/C++. I am a beginner in creating languages and compiler design, I'd like to play around with the source.

For example, theres a hand written lexer on this website before the flex generated lex: http://en.wikipedia.org/wiki/Flex_lexical_analyser

Thanks.

Additional resources that might be of interest: http://stackoverflow.com/questions/1669/learning-to-write-a-compiler — dmckee --- ex-moderator kitten, Oct 05 '09 at 03:45

score 4 · Accepted Answer · answered Oct 05 '09 at 04:16

I like the LLVM tutorial -- it starts by presenting a hand-written lexer for a toy language called Kaleidoscope, first as a "literate programming" exercise with good textual explanations of each bit; it continues by building a parser that builds an abstract syntax tree in the same style; then the whole thing (400 lines of hand-writted standalone C++ for lexer, parser and AST builder) is shown again as a complete .cpp source file, so you don't have to put it all together by yourself. Following chapters show code generation, JIT, optimization, ... a truly useful little tutorial indeed! But, you can stop with the lexer, if that's all you want to understand for now;-).

score 0 · Answer 2 · edited Nov 20 '17 at 16:13

0

You probably want flex, which is probably the most widely used lexer on the planet.

edited Nov 20 '17 at 16:13

rici

219,119
25
219
314

answered Oct 05 '09 at 03:18

timdev

60,131
6
78
90

Looking for hand-written lexers. – Shawn Mclean Oct 05 '09 at 03:20
Oh, missed that bit. Ummmm... good luck, I guess. Is this a homework-related question? I can't easily think of a reason anyone would write a lexer by hand other than they were taking a compiler construction course and it was an assignment. Come to think of it, that sort of assignment would come about right now in the semester. – timdev Oct 05 '09 at 03:26
@Tim, don't you just *hate* it when people repeat themselves like that (as if we couldn't understand). – pavium Oct 05 '09 at 03:29

score 0 · Answer 3 · answered Oct 05 '09 at 03:18

0

Lex and Yacc are the "classic" tools used for the purpose of building languages and compilers.

answered Oct 05 '09 at 03:18

bobbymcr

23,123
3
53
66

Looking for hand-written lexers – Shawn Mclean Oct 05 '09 at 03:22

Free Lexical Analyzer / Scanner Source

3 Answers3