Lrpar codegen by ratmice · Pull Request #657 · softdevteam/grmtools

ratmice · 2026-08-07T03:52:18Z

I think this is probably as reviewable as I'm going to manage to make this large of a patch.
The basic idea behind this patch is to have a kind of "functional pipeline", for doing code generation,

(SrcEnv?, BuildEnvArgs) -> BuildEnv? -> Codegen(SrcEnv, BuildEnv) -> rust_code?

A bit of an oversimplification, as there are some other minor details...

BuildEnvArgs is kind of a minimalist equivalent of the current builder, it's just full of Option values.
BuildEnv is full of derived values, it mostly strips off the Option, but it also contains values like ASTWithValidityInfo that are derived from all the other args.
Codegen then owns the YaccGrammar, StateTable and StateGraphs, which you can take ownership of after generating code.

This may not be totally perfect basis for traits and external usage, for example it currently returns Box<dyn Error> instead of typed errors. But it should be a pretty faithful conversion of the existing process, into a more targeted/self contained module.

…truct.

ltratt · 2026-08-07T08:19:47Z

+    src: &'a str,
+    // We store the path here so we can generate a module name from it if needed.
+    // But should never use it for filesystem interaction within this module.
+    path: &'a Path,


I think that in some future use-cases we might not even have a Path for a given input (e.g. if it's from stdin or a buffer in memory)? If so, I wonder if we should just make this Option<String> for the path? [Maybe not for this PR, but as a simple follow-up.] That might avoid a (borderline-but-not-quite pathological) case where the Path disappears and we don't get a valid pathname out of it later.

I think we kind of need something here stdin, e.g. we derive the default mod_name from it, and error formatting will need some label for error printing, I do think it could be a string though.

e.g. often in the case of stdin this will just be the text "stdin" or something to that effect, for instance the gnu assembler will use the text "standard input"

$ echo bad_instruction | as - Assembler messages: {standard input}:1: Error: no such instruction: `bad_instruction'

Part of the difficulty with turning it into string then is also that SpannedDiagnosticFormatter::new() accepts a path, but it seems like maybe despite it's convenience the diagnostic formatter might not actually belong in the SrcEnv, if we just derive a module name from it we should be able to turn it into a string.

Sorry, I wasn't very clear! What I mean is: if we only use this for a later mod name, we could create the mod name here, rather than carry around a Path.

That makes sense, so once we've removed SpannedDiagnosticsFormatter I think that should work!

ltratt · 2026-08-07T08:37:37Z

+        }
+    }
+
+    fn extract_ast_validation(


Any reason to prefix these methods with extract_? I'm not saying it's wrong, but I wondered if it has more semantic implication than is immediately obvious to me. I also suspect these should document that they mark headers as used?

I had meant to rename these resolve_ but I forgot, is resolve any better?
Essentially these are resolving them to some final/absolute value, from all the possible ways they could have defaults set for them.

I went ahead and renamed these resolve_* in e53376e and docs added in c452dd4.

Let me know if you prefer/can think of a better naming scheme

ltratt · 2026-08-07T08:38:06Z

+        }
+    }
+
+    pub(crate) fn build_env<LexerTypesT>(


Dumb question: could/should this ever be called more than once? If "no" should this consume self?

I think calling it more than once would be a waste and inefficient, there is seems no actual harm in it though.
(I should say, there is no reason to, but there is no reason it couldn't either)

However his can't really consume self though:
Codegen::generate() and many functions (all error handling) after build_env require a &ParserSrcEnv reference. I didn't really think about this, and whether BuildEnv should take ownership of the SrcEnv after the call to build_env though.

FWIW, it looks like Codegen::generate also uses it for error handling.
One thing to note though is what I said in the PR description:

This may not be totally perfect basis for traits and external usage, for example it currently returns Box instead of typed errors.

As an example nimbleparse_lsp wants to send spans directly to the editor, so it wants a totally separate error formatting code than tools like nimbleparse or CTParserBuilder.

Perhaps that changes things.

So, it appears that not all of these errors actually have typed errors (aren't coming from a YaccGrammarError).

For instance the following error, I'm not exactly certain why this error is only being caught so late.
It's the only one that can happen during generate(...), and so is why we need to pass in a src_env.

grmtools/lrpar/src/lib/ctbuilder.rs

Lines 1659 to 1663 in 3a9d88f

s.push_str(&diag.underline_span_with_text(

rule_span,

"Rule missing action type".to_string(),

'^',

));

I'm assuming if we can't catch it earlier for some reason, we could always add a CodegenError type?

I think that the reason this is caught so late, was that long ago GrammarAST could be constructed manually (without a YaccKind), and you could create a YaccGrammar from that. Thus there was nowhere to know whether or not an action type was required or not, because by that point the yacc_kind was lost.

There seems multiple places we could do it, but I might gravitate towards passing the YaccKind in as a parameter to complete_and_validate? Though perhaps not, I don't immediately see how to get the actiontype from the ast, ~~because the YaccParser.global_actiontype is outside the AST~~. There is the Rule.actiont, which global_actiontype gets cloned into for YaccKind::Original, and part of the rule syntax for YaccKind::Grmtools.

FWIW, there was more error handling in gen_user_actions I hadn't noticed regarding action code like $var parsing during gen_user_actions.

I do think it'll probably end up needing a enum CodegenError, just for other kinds of errors though
for separating YaccGrammarError from those originating from proc_macro2, etc.
So I don't know if we absolutely need to catch these earlier?

I've confused myself as to whether this could be self now! :)

I don't think we can quite yet, still working on it. Got to move all the yacc_diag() back out into CTParserBuilder, and make sure there isn't any more yacc_diag() usage inside codegen.
We've gotten rid of the major impediment I've looked at so far though.

ltratt · 2026-08-07T08:40:16Z

+        &self.stable
+    }
+
+    pub(crate) fn take_parser(


This one's interesting because it consumes self. Why/when do we use it in that way?

This is for the return values to CTParserBuilder::build we construct a CTParser from it in the Ok case.
And I think we need to take ownership of the StateTable and StateGraph for CTConflictsError in some of the error cases too.

I was wondering if fn finish(self) -> (YaccGrammar, StateGraph, StateTable) or finished might be a better
name, since technically it's taking more than just the parser?

I don't know if it's really finishing or finalizing anything, but maybe it conveys better that we're done with self and taking ownership of all the interesting values.

ratmice added 14 commits August 6, 2026 19:00

Initial steps towards a lrpar::codegen module.

5a0f5f2

Build the grm, state graph, and state table as part of the code_gen s…

dd71ab0

…truct.

Prepare rebuild_cache to be moved

d51e3b7

Move rebuild_cache to the codegen module

b8c0755

Prepare output_file and friends to be moved

653a285

Move gen_user_actions to the codegen module

1c6e6ac

Move gen_rule_consts to the codegen module

9690e11

Move gen_token_epp to codegen module

7c7f428

Move gen_parse_function to the codegen module

2d95ae4

Move gen_wrappers to codegen module

a521c26

Move most of output_file to the codegen module

6a39e1b

Move additional codegen helpers

b55acef

Remove extraneous pub(crate) since everything is moved

22b6d7f

Remove/fix any lint suppressions

cf24f1d

ratmice assigned ltratt Aug 7, 2026

ltratt reviewed Aug 7, 2026

View reviewed changes

ratmice added 5 commits August 7, 2026 20:10

Use ast_with_validity_info uniformly

b29e3a4

Rename new_with_defaults to new_with_header

c0330c6

Rename extract_* resolve_*

e53376e

manually deindent macro call

9694268

Add docs for resolve methods

c452dd4

	s.push_str(&diag.underline_span_with_text(
	rule_span,
	"Rule missing action type".to_string(),
	'^',
	));

Uh oh!

Conversation

ratmice commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ratmice Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ratmice Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ratmice Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ratmice Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ratmice Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ratmice commented Aug 7, 2026 •

edited

Loading

ratmice Aug 7, 2026 •

edited

Loading

ratmice Aug 8, 2026 •

edited

Loading

ratmice Aug 7, 2026 •

edited

Loading

ratmice Aug 8, 2026 •

edited

Loading

ratmice Aug 8, 2026 •

edited

Loading