This is flex.info, produced by makeinfo version 4.5 from flex.texi.
INFO-DIR-SECTION Programming
START-INFO-DIR-ENTRY
* flex: (flex). Fast lexical analyzer generator (lex replacement).
END-INFO-DIR-ENTRY
The flex manual is placed under the same licensing conditions as the
rest of flex:
Copyright (C) 1990, 1997 The Regents of the University of California.
All rights reserved.
This code is derived from software contributed to Berkeley by Vern
Paxson.
The United States Government has rights in this work pursuant to
contract no. DE-AC03-76SF00098 between the United States Department of
Energy and the University of California.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the
distribution.
Neither the name of the University nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
File: flex.info, Node: unnamed-faq-76, Next: unnamed-faq-77, Prev: unnamed-faq-75, Up: FAQ
unnamed-faq-76
==============
To: "Frescatore, David (CRD, TAD)" <frescatore@exc01crdge.crd.ge.com>
Subject: Re: FLEX 2.5 & THE YEAR 2000
In-reply-to: Your message of Wed, 03 Jun 1998 11:26:22 PDT.
Date: Wed, 03 Jun 1998 10:22:26 PDT
From: Vern Paxson <vern>
> I am researching the Y2K problem with General Electric R&D
> and need to know if there are any known issues concerning
> the above mentioned software and Y2K regardless of version.
There shouldn't be, all it ever does with the date is ask the system
for it and then print it out.
Vern
File: flex.info, Node: unnamed-faq-77, Next: unnamed-faq-78, Prev: unnamed-faq-76, Up: FAQ
unnamed-faq-77
==============
To: "Hans Dermot Doran" <htd@ibhdoran.com>
Subject: Re: flex problem
In-reply-to: Your message of Wed, 15 Jul 1998 21:30:13 PDT.
Date: Tue, 21 Jul 1998 14:23:34 PDT
From: Vern Paxson <vern>
> To overcome this, I gets() the stdin into a string and lex the string. The
> string is lexed OK except that the end of string isn't lexed properly
> (yy_scan_string()), that is the lexer dosn't recognise the end of string.
Flex doesn't contain mechanisms for recognizing buffer endpoints. But if
you use fgets instead (which you should anyway, to protect against buffer
overflows), then the final \n will be preserved in the string, and you can
scan that in order to find the end of the string.
Vern
File: flex.info, Node: unnamed-faq-78, Next: unnamed-faq-79, Prev: unnamed-faq-77, Up: FAQ
unnamed-faq-78
==============
To: soumen@almaden.ibm.com
Subject: Re: Flex++ 2.5.3 instance member vs. static member
In-reply-to: Your message of Mon, 27 Jul 1998 02:10:04 PDT.
Date: Tue, 28 Jul 1998 01:10:34 PDT
From: Vern Paxson <vern>
> %{
> int mylineno = 0;
> %}
> ws [ \t]+
> alpha [A-Za-z]
> dig [0-9]
> %%
>
> Now you'd expect mylineno to be a member of each instance of class
> yyFlexLexer, but is this the case? A look at the lex.yy.cc file seems to
> indicate otherwise; unless I am missing something the declaration of
> mylineno seems to be outside any class scope.
>
> How will this work if I want to run a multi-threaded application with each
> thread creating a FlexLexer instance?
Derive your own subclass and make mylineno a member variable of it.
Vern
File: flex.info, Node: unnamed-faq-79, Next: unnamed-faq-80, Prev: unnamed-faq-78, Up: FAQ
unnamed-faq-79
==============
To: Adoram Rogel <adoram@hybridge.com>
Subject: Re: More than 32K states change hangs
In-reply-to: Your message of Tue, 04 Aug 1998 16:55:39 PDT.
Date: Tue, 04 Aug 1998 22:28:45 PDT
From: Vern Paxson <vern>
> Vern Paxson,
>
> I followed your advice, posted on Usenet bu you, and emailed to me
> personally by you, on how to overcome the 32K states limit. I'm running
> on Linux machines.
> I took the full source of version 2.5.4 and did the following changes in
> flexdef.h:
> #define JAMSTATE -327660
> #define MAXIMUM_MNS 319990
> #define BAD_SUBSCRIPT -327670
> #define MAX_SHORT 327000
>
> and compiled.
> All looked fine, including check and bigcheck, so I installed.
Hmmm, you shouldn't increase MAX_SHORT, though looking through my email
archives I see that I did indeed recommend doing so. Try setting it back
to 32700; that should suffice that you no longer need -Ca. If it still
hangs, then the interesting question is - where?
> Compiling the same hanged program with a out-of-the-box (RedHat 4.2
> distribution of Linux)
> flex 2.5.4 binary works.
Since Linux comes with source code, you should diff it against what
you have to see what problems they missed.
> Should I always compile with the -Ca option now ? even short and simple
> filters ?
No, definitely not. It's meant to be for those situations where you
absolutely must squeeze every last cycle out of your scanner.
Vern
File: flex.info, Node: unnamed-faq-80, Next: unnamed-faq-81, Prev: unnamed-faq-79, Up: FAQ
unnamed-faq-80
==============
To: "Schmackpfeffer, Craig" <Craig.Schmackpfeffer@usa.xerox.com>
Subject: Re: flex output for static code portion
In-reply-to: Your message of Tue, 11 Aug 1998 11:55:30 PDT.
Date: Mon, 17 Aug 1998 23:57:42 PDT
From: Vern Paxson <vern>
> I would like to use flex under the hood to generate a binary file
> containing the data structures that control the parse.
This has been on the wish-list for a long time. In principle it's
straight-forward - you redirect mkdata() et al's I/O to another file,
and modify the skeleton to have a start-up function that slurps these
into dynamic arrays. The concerns are (1) the scanner generation code
is hairy and full of corner cases, so it's easy to get surprised when
going down this path :-( ; and (2) being careful about buffering so
that when the tables change you make sure the scanner starts in the
correct state and reading at the right point in the input file.
> I was wondering if you know of anyone who has used flex in this way.
I don't - but it seems like a reasonable project to undertake (unlike
numerous other flex tweaks :-).
Vern
File: flex.info, Node: unnamed-faq-81, Next: unnamed-faq-82, Prev: unnamed-faq-80, Up: FAQ
unnamed-faq-81
==============
Received: from 131.173.17.11 (131.173.17.11 [131.173.17.11])
by ee.lbl.gov (8.9.1/8.9.1) with ESMTP id AAA03838
for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 00:47:57 -0700 (PDT)
Received: from hal.cl-ki.uni-osnabrueck.de (hal.cl-ki.Uni-Osnabrueck.DE [131.173.141.2])
by deimos.rz.uni-osnabrueck.de (8.8.7/8.8.8) with ESMTP id JAA34694
for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 09:47:55 +0200
Received: (from georg@localhost) by hal.cl-ki.uni-osnabrueck.de (8.6.12/8.6.12) id JAA34834 for vern@ee.lbl.gov; Thu, 20 Aug 1998 09:47:54 +0200
From: Georg Rehm <georg@hal.cl-ki.uni-osnabrueck.de>
Message-Id: <199808200747.JAA34834@hal.cl-ki.uni-osnabrueck.de>
Subject: "flex scanner push-back overflow"
To: vern@ee.lbl.gov
Date: Thu, 20 Aug 1998 09:47:54 +0200 (MEST)
Reply-To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE
X-NoJunk: Do NOT send commercial mail, spam or ads to this address!
X-URL: http://www.cl-ki.uni-osnabrueck.de/~georg/
X-Mailer: ELM [version 2.4ME+ PL28 (25)]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Hi Vern,
Yesterday, I encountered a strange problem: I use the macro processor m4
to include some lengthy lists into a .l file. Following is a flex macro
definition that causes some serious pain in my neck:
AUTHOR ("A. Boucard / L. Boucard"|"A. Dastarac / M. Levent"|"A.Boucaud / L.Boucaud"|"Abderrahim Lamchichi"|"Achmat Dangor"|"Adeline Toullier"|"Adewale Maja-Pearce"|"Ahmed Ziri"|"Akram Ellyas"|"Alain Bihr"|"Alain Gresh"|"Alain Guillemoles"|"Alain Joxe"|"Alain Morice"|"Alain Renon"|"Alain Zecchini"|"Albert Memmi"|"Alberto Manguel"|"Alex De Waal"|"Alfonso Artico"| [...])
The complete list contains about 10kB. When I try to "flex" this file
(on a Solaris 2.6 machine, using a modified flex 2.5.4 (I only increased
some of the predefined values in flexdefs.h) I get the error:
myflex/flex -8 sentag.tmp.l
flex scanner push-back overflow
When I remove the slashes in the macro definition everything works fine.
As I understand it, the double quotes escape the slash-character so it
really means "/" and not "trailing context". Furthermore, I tried to
escape the slashes with backslashes, but with no use, the same error message
appeared when flexing the code.
Do you have an idea what's going on here?
Greetings from Germany,
Georg
--
Georg Rehm georg@cl-ki.uni-osnabrueck.de
Institute for Semantic Information Processing, University of Osnabrueck, FRG
File: flex.info, Node: unnamed-faq-82, Next: unnamed-faq-83, Prev: unnamed-faq-81, Up: FAQ
unnamed-faq-82
==============
To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE
Subject: Re: "flex scanner push-back overflow"
In-reply-to: Your message of Thu, 20 Aug 1998 09:47:54 PDT.
Date: Thu, 20 Aug 1998 07:05:35 PDT
From: Vern Paxson <vern>
> myflex/flex -8 sentag.tmp.l
> flex scanner push-back overflow
Flex itself uses a flex scanner. That scanner is running out of buffer
space when it tries to unput() the humongous macro you've defined. When
you remove the '/'s, you make it small enough so that it fits in the buffer;
removing spaces would do the same thing.
The fix is to either rethink how come you're using such a big macro and
perhaps there's another/better way to do it; or to rebuild flex's own
scan.c with a larger value for
#define YY_BUF_SIZE 16384
- Vern
File: flex.info, Node: unnamed-faq-83, Next: unnamed-faq-84, Prev: unnamed-faq-82, Up: FAQ
unnamed-faq-83
==============
To: Jan Kort <jan@research.techforce.nl>
Subject: Re: Flex
In-reply-to: Your message of Fri, 04 Sep 1998 12:18:43 +0200.
Date: Sat, 05 Sep 1998 00:59:49 PDT
From: Vern Paxson <vern>
> %%
>
> "TEST1\n" { fprintf(stderr, "TEST1\n"); yyless(5); }
> ^\n { fprintf(stderr, "empty line\n"); }
> . { }
> \n { fprintf(stderr, "new line\n"); }
>
> %%
> -- input ---------------------------------------
> TEST1
> -- output --------------------------------------
> TEST1
> empty line
> ------------------------------------------------
IMHO, it's not clear whether or not this is in fact a bug. It depends
on whether you view yyless() as backing up in the input stream, or as
pushing new characters onto the beginning of the input stream. Flex
interprets it as the latter (for implementation convenience, I'll admit),
and so considers the newline as in fact matching at the beginning of a
line, as after all the last token scanned an entire line and so the
scanner is now at the beginning of a new line.
I agree that this is counter-intuitive for yyless(), given its
functional description (it's less so for unput(), depending on whether
you're unput()'ing new text or scanned text). But I don't plan to
change it any time soon, as it's a pain to do so. Consequently,
you do indeed need to use yy_set_bol() and YY_AT_BOL() to tweak
your scanner into the behavior you desire.
Sorry for the less-than-completely-satisfactory answer.
Vern
File: flex.info, Node: unnamed-faq-84, Next: unnamed-faq-85, Prev: unnamed-faq-83, Up: FAQ
unnamed-faq-84
==============
To: Patrick Krusenotto <krusenot@mac-info-link.de>
Subject: Re: Problems with restarting flex-2.5.2-generated scanner
In-reply-to: Your message of Thu, 24 Sep 1998 10:14:07 PDT.
Date: Thu, 24 Sep 1998 23:28:43 PDT
From: Vern Paxson <vern>
> I am using flex-2.5.2 and bison 1.25 for Solaris and I am desperately
> trying to make my scanner restart with a new file after my parser stops
> with a parse error. When my compiler restarts, the parser always
> receives the token after the token (in the old file!) that caused the
> parser error.
I suspect the problem is that your parser has read ahead in order
to attempt to resolve an ambiguity, and when it's restarted it picks
up with that token rather than reading a fresh one. If you're using
yacc, then the special "error" production can sometimes be used to
consume tokens in an attempt to get the parser into a consistent state.
Vern
File: flex.info, Node: unnamed-faq-85, Next: unnamed-faq-86, Prev: unnamed-faq-84, Up: FAQ
unnamed-faq-85
==============
To: Henric Jungheim <junghelh@pe-nelson.com>
Subject: Re: flex 2.5.4a
In-reply-to: Your message of Tue, 27 Oct 1998 16:41:42 PST.
Date: Tue, 27 Oct 1998 16:50:14 PST
From: Vern Paxson <vern>
> This brings up a feature request: How about a command line
> option to specify the filename when reading from stdin? That way one
> doesn't need to create a temporary file in order to get the "#line"
> directives to make sense.
Use -o combined with -t (per the man page description of -o).
> P.S., Is there any simple way to use non-blocking IO to parse multiple
> streams?
Simple, no.
One approach might be to return a magic character on EWOULDBLOCK and
have a rule
.*<magic-character> // put back .*, eat magic character
This is off the top of my head, not sure it'll work.
Vern
File: flex.info, Node: unnamed-faq-86, Next: unnamed-faq-87, Prev: unnamed-faq-85, Up: FAQ
unnamed-faq-86
==============
To: "Repko, Billy D" <billy.d.repko@intel.com>
Subject: Re: Compiling scanners
In-reply-to: Your message of Wed, 13 Jan 1999 10:52:47 PST.
Date: Thu, 14 Jan 1999 00:25:30 PST
From: Vern Paxson <vern>
> It appears that maybe it cannot find the lfl library.
The Makefile in the distribution builds it, so you should have it.
It's exceedingly trivial, just a main() that calls yylex() and
a yyrap() that always returns 1.
> %%
> \n ++num_lines; ++num_chars;
> . ++num_chars;
You can't indent your rules like this - that's where the errors are coming
from. Flex copies indented text to the output file, it's how you do things
like
int num_lines_seen = 0;
to declare local variables.
Vern
File: flex.info, Node: unnamed-faq-87, Next: unnamed-faq-88, Prev: unnamed-faq-86, Up: FAQ
unnamed-faq-87
==============
To: Erick Branderhorst <Erick.Branderhorst@asml.nl>
Subject: Re: flex input buffer
In-reply-to: Your message of Tue, 09 Feb 1999 13:53:46 PST.
Date: Tue, 09 Feb 1999 21:03:37 PST
From: Vern Paxson <vern>
> In the flex.skl file the size of the default input buffers is set. Can you
> explain why this size is set and why it is such a high number.
It's large to optimize performance when scanning large files. You can
safely make it a lot lower if needed.
Vern
File: flex.info, Node: unnamed-faq-88, Next: unnamed-faq-90, Prev: unnamed-faq-87, Up: FAQ
unnamed-faq-88
==============
To: "Guido Minnen" <guidomi@cogs.susx.ac.uk>
Subject: Re: Flex error message
In-reply-to: Your message of Wed, 24 Feb 1999 15:31:46 PST.
Date: Thu, 25 Feb 1999 00:11:31 PST
From: Vern Paxson <vern>
> I'm extending a larger scanner written in Flex and I keep running into
> problems. More specifically, I get the error message:
> "flex: input rules are too complicated (>= 32000 NFA states)"
Increase the definitions in flexdef.h for:
#define JAMSTATE -32766 /* marks a reference to the state that always j
ams */
#define MAXIMUM_MNS 31999
#define BAD_SUBSCRIPT -32767
recompile everything, and it should all work.
Vern
File: flex.info, Node: unnamed-faq-90, Next: unnamed-faq-91, Prev: unnamed-faq-88, Up: FAQ
unnamed-faq-90
==============
To: "Dmitriy Goldobin" <gold@ems.chel.su>
Subject: Re: FLEX trouble
In-reply-to: Your message of Mon, 31 May 1999 18:44:49 PDT.
Date: Tue, 01 Jun 1999 00:15:07 PDT
From: Vern Paxson <vern>
> I have a trouble with FLEX. Why rule "/*".*"*/" work properly,=20
> but rule "/*"(.|\n)*"*/" don't work ?
The second of these will have to scan the entire input stream (because
"(.|\n)*" matches an arbitrary amount of any text) in order to see if
it ends with "*/", terminating the comment. That potentially will overflow
the input buffer.
> More complex rule "/*"([^*]|(\*/[^/]))*"*/ give an error
> 'unrecognized rule'.
You can't use the '/' operator inside parentheses. It's not clear
what "(a/b)*" actually means.
> I now use workaround with state <comment>, but single-rule is
> better, i think.
Single-rule is nice but will always have the problem of either setting
restrictions on comments (like not allowing multi-line comments) and/or
running the risk of consuming the entire input stream, as noted above.
Vern
File: flex.info, Node: unnamed-faq-91, Next: unnamed-faq-92, Prev: unnamed-faq-90, Up: FAQ
unnamed-faq-91
==============
Received: from mc-qout4.whowhere.com (mc-qout4.whowhere.com [209.185.123.18])
by ee.lbl.gov (8.9.3/8.9.3) with SMTP id IAA05100
for <vern@ee.lbl.gov>; Tue, 15 Jun 1999 08:56:06 -0700 (PDT)
Received: from Unknown/Local ([?.?.?.?]) by my-deja.com; Tue Jun 15 08:55:43 1999
To: vern@ee.lbl.gov
Date: Tue, 15 Jun 1999 08:55:43 -0700
From: "Aki Niimura" <neko@my-deja.com>
Message-ID: <KNONDOHDOBGAEAAA@my-deja.com>
Mime-Version: 1.0
Cc:
X-Sent-Mail: on
Reply-To:
X-Mailer: MailCity Service
Subject: A question on flex C++ scanner
X-Sender-Ip: 12.72.207.61
Organization: My Deja Email (http://www.my-deja.com:80)
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Dear Dr. Paxon,
I have been using flex for years.
It works very well on many projects.
Most case, I used it to generate a scanner on C language.
However, one project I needed to generate a scanner
on C++ lanuage. Thanks to your enhancement, flex did
the job.
Currently, I'm working on enhancing my previous project.
I need to deal with multiple input streams (recursive
inclusion) in this scanner (C++).
I did similar thing for another scanner (C) as you
explained in your documentation.
The generated scanner (C++) has necessary methods:
- switch_to_buffer(struct yy_buffer_state *b)
- yy_create_buffer(istream *is, int sz)
- yy_delete_buffer(struct yy_buffer_state *b)
However, I couldn't figure out how to access current
buffer (yy_current_buffer).
yy_current_buffer is a protected member of yyFlexLexer.
I can't access it directly.
Then, I thought yy_create_buffer() with is = 0 might
return current stream buffer. But it seems not as far
as I checked the source. (flex 2.5.4)
I went through the Web in addition to Flex documentation.
However, it hasn't been successful, so far.
It is not my intention to bother you, but, can you
comment about how to obtain the current stream buffer?
Your response would be highly appreciated.
Best regards,
Aki Niimura
--== Sent via Deja.com http://www.deja.com/ ==--
Share what you know. Learn what you don't.
File: flex.info, Node: unnamed-faq-92, Next: unnamed-faq-93, Prev: unnamed-faq-91, Up: FAQ
unnamed-faq-92
==============
To: neko@my-deja.com
Subject: Re: A question on flex C++ scanner
In-reply-to: Your message of Tue, 15 Jun 1999 08:55:43 PDT.
Date: Tue, 15 Jun 1999 09:04:24 PDT
From: Vern Paxson <vern>
> However, I couldn't figure out how to access current
> buffer (yy_current_buffer).
Derive your own subclass from yyFlexLexer.
Vern
File: flex.info, Node: unnamed-faq-93, Next: unnamed-faq-94, Prev: unnamed-faq-92, Up: FAQ
unnamed-faq-93
==============
To: "Stones, Darren" <Darren.Stones@nectech.co.uk>
Subject: Re: You're the man to see?
In-reply-to: Your message of Wed, 23 Jun 1999 11:10:29 PDT.
Date: Wed, 23 Jun 1999 09:01:40 PDT
From: Vern Paxson <vern>
> I hope you can help me. I am using Flex and Bison to produce an interpreted
> language. However all goes well until I try to implement an IF statement or
> a WHILE. I cannot get this to work as the parser parses all the conditions
> eg. the TRUE and FALSE conditons to check for a rule match. So I cannot
> make a decision!!
You need to use the parser to build a parse tree (= abstract syntax trwee),
and when that's all done you recursively evaluate the tree, binding variables
to values at that time.
Vern
File: flex.info, Node: unnamed-faq-94, Next: unnamed-faq-95, Prev: unnamed-faq-93, Up: FAQ
unnamed-faq-94
==============
To: Petr Danecek <petr@ics.cas.cz>
Subject: Re: flex - question
In-reply-to: Your message of Mon, 28 Jun 1999 19:21:41 PDT.
Date: Fri, 02 Jul 1999 16:52:13 PDT
From: Vern Paxson <vern>
> file, it takes an enormous amount of time. It is funny, because the
> source code has only 12 rules!!! I think it looks like an exponencial
> growth.
Right, that's the problem - some patterns (those with a lot of
ambiguity, where yours has because at any given time the scanner can
be in the middle of all sorts of combinations of the different
rules) blow up exponentially.
For your rules, there is an easy fix. Change the ".*" that comes fater
the directory name to "[^ ]*". With that in place, the rules are no
longer nearly so ambiguous, because then once one of the directories
has been matched, no other can be matched (since they all require a
leading blank).
If that's not an acceptable solution, then you can enter a start state
to pick up the .*\n after each directory is matched.
Also note that for speed, you'll want to add a ".*" rule at the end,
otherwise rules that don't match any of the patterns will be matched
very slowly, a character at a time.
Vern
File: flex.info, Node: unnamed-faq-95, Next: unnamed-faq-96, Prev: unnamed-faq-94, Up: FAQ
unnamed-faq-95
==============
To: Tielman Koekemoer <tielman@spi.co.za>
Subject: Re: Please help.
In-reply-to: Your message of Thu, 08 Jul 1999 13:20:37 PDT.
Date: Thu, 08 Jul 1999 08:20:39 PDT
From: Vern Paxson <vern>
> I was hoping you could help me with my problem.
>
> I tried compiling (gnu)flex on a Solaris 2.4 machine
> but when I ran make (after configure) I got an error.
>
> --------------------------------------------------------------
> gcc -c -I. -I. -g -O parse.c
> ./flex -t -p ./scan.l >scan.c
> sh: ./flex: not found
> *** Error code 1
> make: Fatal error: Command failed for target `scan.c'
> -------------------------------------------------------------
>
> What's strange to me is that I'm only
> trying to install flex now. I then edited the Makefile to
> and changed where it says "FLEX = flex" to "FLEX = lex"
> ( lex: the native Solaris one ) but then it complains about
> the "-p" option. Is there any way I can compile flex without
> using flex or lex?
>
> Thanks so much for your time.
You managed to step on the bootstrap sequence, which first copies
initscan.c to scan.c in order to build flex. Try fetching a fresh
distribution from ftp.ee.lbl.gov. (Or you can first try removing
".bootstrap" and doing a make again.)
Vern
File: flex.info, Node: unnamed-faq-96, Next: unnamed-faq-97, Prev: unnamed-faq-95, Up: FAQ
unnamed-faq-96
==============
To: Tielman Koekemoer <tielman@spi.co.za>
Subject: Re: Please help.
In-reply-to: Your message of Fri, 09 Jul 1999 09:16:14 PDT.
Date: Fri, 09 Jul 1999 00:27:20 PDT
From: Vern Paxson <vern>
> First I removed .bootstrap (and ran make) - no luck. I downloaded the
> software but I still have the same problem. Is there anything else I
> could try.
Try:
cp initscan.c scan.c
touch scan.c
make scan.o
If this last tries to first build scan.c from scan.l using ./flex, then
your "make" is broken, in which case compile scan.c to scan.o by hand.
Vern
File: flex.info, Node: unnamed-faq-97, Next: unnamed-faq-98, Prev: unnamed-faq-96, Up: FAQ
unnamed-faq-97
==============
To: Sumanth Kamenani <skamenan@crl.nmsu.edu>
Subject: Re: Error
In-reply-to: Your message of Mon, 19 Jul 1999 23:08:41 PDT.
Date: Tue, 20 Jul 1999 00:18:26 PDT
From: Vern Paxson <vern>
> I am getting a compilation error. The error is given as "unknown symbol- yylex".
The parser relies on calling yylex(), but you're instead using the C++ scanning
class, so you need to supply a yylex() "glue" function that calls an instance
scanner of the scanner (e.g., "scanner->yylex()").
Vern
File: flex.info, Node: unnamed-faq-98, Next: unnamed-faq-99, Prev: unnamed-faq-97, Up: FAQ
unnamed-faq-98
==============
To: daniel@synchrods.synchrods.COM (Daniel Senderowicz)
Subject: Re: lex
In-reply-to: Your message of Mon, 22 Nov 1999 11:19:04 PST.
Date: Tue, 23 Nov 1999 15:54:30 PST
From: Vern Paxson <vern>
Well, your problem is the
switch (yybgin-yysvec-1) { /* witchcraft */
at the beginning of lex rules. "witchcraft" == "non-portable". It's
assuming knowledge of the AT&T lex's internal variables.
For flex, you can probably do the equivalent using a switch on YYSTATE.
Vern
File: flex.info, Node: unnamed-faq-99, Next: unnamed-faq-100, Prev: unnamed-faq-98, Up: FAQ
unnamed-faq-99
==============
To: archow@hss.hns.com
Subject: Re: Regarding distribution of flex and yacc based grammars
In-reply-to: Your message of Sun, 19 Dec 1999 17:50:24 +0530.
Date: Wed, 22 Dec 1999 01:56:24 PST
From: Vern Paxson <vern>
> When we provide the customer with an object code distribution, is it
> necessary for us to provide source
> for the generated C files from flex and bison since they are generated by
> flex and bison ?
For flex, no. I don't know what the current state of this is for bison.
> Also, is there any requrirement for us to neccessarily provide source for
> the grammar files which are fed into flex and bison ?
Again, for flex, no.
See the file "COPYING" in the flex distribution for the legalese.
Vern
File: flex.info, Node: unnamed-faq-100, Next: unnamed-faq-101, Prev: unnamed-faq-99, Up: FAQ
unnamed-faq-100
===============
To: Martin Gallwey <gallweym@hyperion.moe.ul.ie>
Subject: Re: Flex, and self referencing rules
In-reply-to: Your message of Sun, 20 Feb 2000 01:01:21 PST.
Date: Sat, 19 Feb 2000 18:33:16 PST
From: Vern Paxson <vern>
> However, I do not use unput anywhere. I do use self-referencing
> rules like this:
>
> UnaryExpr ({UnionExpr})|("-"{UnaryExpr})
You can't do this - flex is *not* a parser like yacc (which does indeed
allow recursion), it is a scanner that's confined to regular expressions.
Vern
File: flex.info, Node: unnamed-faq-101, Next: What is the difference between YYLEX_PARAM and YY_DECL?, Prev: unnamed-faq-100, Up: FAQ
unnamed-faq-101
===============
To: slg3@lehigh.edu (SAMUEL L. GULDEN)
Subject: Re: Flex problem
In-reply-to: Your message of Thu, 02 Mar 2000 12:29:04 PST.
Date: Thu, 02 Mar 2000 23:00:46 PST
From: Vern Paxson <vern>
If this is exactly your program:
> digit [0-9]
> digits {digit}+
> whitespace [ \t\n]+
>
> %%
> "[" { printf("open_brac\n");}
> "]" { printf("close_brac\n");}
> "+" { printf("addop\n");}
> "*" { printf("multop\n");}
> {digits} { printf("NUMBER = %s\n", yytext);}
> whitespace ;
then the problem is that the last rule needs to be "{whitespace}" !
Vern
File: flex.info, Node: What is the difference between YYLEX_PARAM and YY_DECL?, Next: Why do I get "conflicting types for yylex" error?, Prev: unnamed-faq-101, Up: FAQ
What is the difference between YYLEX_PARAM and YY_DECL?
=======================================================
YYLEX_PARAM is not a flex symbol. It is for Bison. It tells Bison to
pass extra params when it calls yylex() from the parser.
YY_DECL is the Flex declaration of yylex. The default is similar to
this:
#define int yy_lex ()
File: flex.info, Node: Why do I get "conflicting types for yylex" error?, Next: How do I access the values set in a Flex action from within a Bison action?, Prev: What is the difference between YYLEX_PARAM and YY_DECL?, Up: FAQ
Why do I get "conflicting types for yylex" error?
=================================================
This is a compiler error regarding a generated Bison parser, not a
Flex scanner. It means you need a prototype of yylex() in the top of
the Bison file. Be sure the prototype matches YY_DECL.
File: flex.info, Node: How do I access the values set in a Flex action from within a Bison action?, Prev: Why do I get "conflicting types for yylex" error?, Up: FAQ
How do I access the values set in a Flex action from within a Bison action?
===========================================================================
With $1, $2, $3, etc. These are called "Semantic Values" in the
Bison manual. See *Note Top: (bison)Top.
File: flex.info, Node: Appendices, Next: Indices, Prev: FAQ, Up: Top
Appendices
**********
* Menu:
* Makefiles and Flex::
* Bison Bridge::
* M4 Dependency::
File: flex.info, Node: Makefiles and Flex, Next: Bison Bridge, Prev: Appendices, Up: Appendices
Makefiles and Flex
==================
In this appendix, we provide tips for writing Makefiles to build
your scanners.
In a traditional build environment, we say that the `.c' files are
the sources, and the `.o' files are the intermediate files. When using
`flex', however, the `.l' files are the sources, and the generated `.c'
files (along with the `.o' files) are the intermediate files. This
requires you to carefully plan your Makefile.
Modern `make' programs understand that `foo.l' is intended to
generate `lex.yy.c' or `foo.c', and will behave accordingly(1). The
following Makefile does not explicitly instruct `make' how to build
`foo.c' from `foo.l'. Instead, it relies on the implicit rules of the
`make' program to build the intermediate file, `scan.c':
# Basic Makefile -- relies on implicit rules
# Creates "myprogram" from "scan.l" and "myprogram.c"
#
LEX=flex
myprogram: scan.o myprogram.o
scan.o: scan.l
For simple cases, the above may be sufficient. For other cases, you
may have to explicitly instruct `make' how to build your scanner. The
following is an example of a Makefile containing explicit rules:
# Basic Makefile -- provides explicit rules
# Creates "myprogram" from "scan.l" and "myprogram.c"
#
LEX=flex
myprogram: scan.o myprogram.o
$(CC) -o $@ $(LDFLAGS) $^
myprogram.o: myprogram.c
$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^
scan.o: scan.c
$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^
scan.c: scan.l
$(LEX) $(LFLAGS) -o $@ $^
clean:
$(RM) *.o scan.c
Notice in the above example that `scan.c' is in the `clean' target.
This is because we consider the file `scan.c' to be an intermediate
file.
Finally, we provide a realistic example of a `flex' scanner used
with a `bison' parser(2). There is a tricky problem we have to deal
with. Since a `flex' scanner will typically include a header file
(e.g., `y.tab.h') generated by the parser, we need to be sure that the
header file is generated BEFORE the scanner is compiled. We handle this
case in the following example:
# Makefile example -- scanner and parser.
# Creates "myprogram" from "scan.l", "parse.y", and "myprogram.c"
#
LEX = flex
YACC = bison -y
YFLAGS = -d
objects = scan.o parse.o myprogram.o
myprogram: $(objects)
scan.o: scan.l parse.c
parse.o: parse.y
myprogram.o: myprogram.c
In the above example, notice the line,
scan.o: scan.l parse.c
, which lists the file `parse.c' (the generated parser) as a
dependency of `scan.o'. We want to ensure that the parser is created
before the scanner is compiled, and the above line seems to do the
trick. Feel free to experiment with your specific implementation of
`make'.
For more details on writing Makefiles, see *Note Top: (make)Top.
---------- Footnotes ----------
(1) GNU `make' and GNU `automake' are two such programs that provide
implicit rules for flex-generated scanners.
(2) This example also applies to yacc parsers.
File: flex.info, Node: Bison Bridge, Next: M4 Dependency, Prev: Makefiles and Flex, Up: Appendices
C Scanners with Bison Parsers
=============================
This section describes the `flex' features useful when integrating
`flex' with `GNU bison'(1). Skip this section if you are not using
`bison' with your scanner. Here we discuss only the `flex' half of the
`flex' and `bison' pair. We do not discuss `bison' in any detail. For
more information about generating `bison' parsers, see *Note Top:
(bison)Top.
A compatible `bison' scanner is generated by declaring `%option
bison-bridge' or by supplying `--bison-bridge' when invoking `flex'
from the command line. This instructs `flex' that the macro `yylval'
may be used. The data type for `yylval', `YYSTYPE', is typically
defined in a header file, included in section 1 of the `flex' input
file. For a list of functions and macros available, *Note
bison-functions::.
The declaration of yylex becomes,
int yylex ( YYSTYPE * lvalp, yyscan_t scanner );
If `%option bison-locations' is specified, then the declaration
becomes,
int yylex ( YYSTYPE * lvalp, YYLTYPE * llocp, yyscan_t scanner );
Note that the macros `yylval' and `yylloc' evaluate to pointers.
Support for `yylloc' is optional in `bison', so it is optional in
`flex' as well. The following is an example of a `flex' scanner that is
compatible with `bison'.
/* Scanner for "C" assignment statements... sort of. */
%{
#include "y.tab.h" /* Generated by bison. */
%}
%option bison-bridge bison-locations
%
[[:digit:]]+ { yylval->num = atoi(yytext); return NUMBER;}
[[:alnum:]]+ { yylval->str = strdup(yytext); return STRING;}
"="|";" { return yytext[0];}
. {}
%
As you can see, there really is no magic here. We just use `yylval'
as we would any other variable. The data type of `yylval' is generated
by `bison', and included in the file `y.tab.h'. Here is the
corresponding `bison' parser:
/* Parser to convert "C" assignments to lisp. */
%{
/* Pass the argument to yyparse through to yylex. */
#define YYPARSE_PARAM scanner
#define YYLEX_PARAM scanner
%}
%locations
%pure_parser
%union {
int num;
char* str;
}
%token <str> STRING
%token <num> NUMBER
%%
assignment:
STRING '=' NUMBER ';' {
printf( "(setf %s %d)", $1, $3 );
}
;
---------- Footnotes ----------
(1) The features described here are purely optional, and are by no
means the only way to use flex with bison. We merely provide some glue
to ease development of your parser-scanner pair.
File: flex.info, Node: M4 Dependency, Prev: Bison Bridge, Up: Appendices
M4 Dependency
=============
The macro processor `m4'(1) must be installed wherever flex is
installed. `flex' invokes `m4', found by searching the directories in
the `PATH' environment variable. Any code you place in section 1 or in
the actions will be sent through m4. Please follow these rules to
protect your code from unwanted `m4' processing.
* Do not use symbols that begin with, `m4_', such as, `m4_define',
or `m4_include', since those are reserved for `m4' macro names. If
for some reason you need m4_ as a prefix, use a preprocessor
#define to get your symbol past m4 unmangled.
* Do not use the strings `[[' or `]]' anywhere in your code. The
former is not valid in C, except within comments and strings, but
the latter is valid in code such as `x[y[z]]'. The solution is
simple. To get the literal string `"]]"', use `"]""]"'. To get the
array notation `x[y[z]]', use `x[y[z] ]'.
`m4' is only required at the time you run `flex'. The generated
scanner is ordinary C or C++, and does _not_ require `m4'.
---------- Footnotes ----------
(1) The use of m4 is subject to change in future revisions of flex.
It is not part of the public API of flex. Do not depend on it.
File: flex.info, Node: Indices, Prev: Appendices, Up: Top
Indices
*******
* Menu:
* Concept Index::
* Index of Functions and Macros::
* Index of Variables::
* Index of Data Types::
* Index of Hooks::
* Index of Scanner Options::