Скачать или смотреть Compiler From Scratch: Phase 1 - Tokenizer Generator 023: Tokenizing tokenizer config file. Testing.

Streamed on 2024-12-20 (  / thediscouragerofhesitancy  )

Zero Dependencies Programming!

Last week we started down a dark road of multiple-encoding support for configuring the tokenizer generator. We stepped back from the edge and had a plan set for this week's efforts, and it started with undoing some of the things we did on that dark road.

With several of the changes rolled back, we could proceed as planned. The next step was to generate a tokenizer for the tokenizer definition file. While this wasn't too difficult in the general sense, it wasn't trivial, largely due to the very meta nature of this stage. We are using the tokenizer generator to create a tokenizer that will be used by the next build of the tokenizer generator. There was some back and forth to keep all of the code and configuration aligned in such a way that old code/config and the future code/config all remained in agreement. Once a simple test of it was working I used the new tokenizer in the parsing of the tokenizer definition file.

Now that we are built on our own generated code it was time to update tests. File names had changed, file formats had changed, and it wasn't working as before. Fixing the afore mentioned items got us passing all automated tests again. Then I turned to fixing the manual testing build. I got it working again, but broken something in the automated test side again and that's where we ran out of time. I'll finish debugging/fixing that last test bug next week.