flex and bison usage in mysql

query parsing in mysql

mysql source code version: 8.0.34 (from MYSQL_VERSION file)

This an article from questions to understandings.

• which file does mysql use to define sql grammar?
  sql/sql_yacc.yy

• what is the name yyparse replaced with in mysql?
  Searching name-prefix in mysql source code, you will get --name-prefix=MYSQL in sql/CMakeLists.txt. In short, yyparse function is replaced with MYSQLparse function. In latest version of mysql, yyparse is named as my_sql_parser_parse due to the directive %define api.prefix {my_sql_parser_}.

• where does the MYSQLparse function be called?
  In THD::sql_parser() method, which is defined in sql/sql_class.cc file.

• what is the signature of MYSQLparse function and how is it generated?
  Here's the signature,

  extern int MYSQLparse(class THD * thd, class Parse_tree_root * *root);
  

  This signature will be generated by bison command with following declaration:

  %parse-param { class THD *YYTHD }
  %parse-param { class Parse_tree_root **parse_tree }
  

• How does the parameters YYTHD and parse_tree be used in semantic actions?
  Here's how YYTHD used,

  prepare:
        PREPARE_SYM ident FROM prepare_src
        {
          THD *thd= YYTHD;
          LEX *lex= thd->lex;
          lex->sql_command= SQLCOM_PREPARE;
          lex->prepared_stmt_name= to_lex_cstring($2);
          lex->contains_plaintext_password= true;
        }
      ;
  
  

  Here's how parse_tree used,

  simple_statement_or_begin:
        simple_statement      { *parse_tree= $1; }
      | begin_stmt
      ;
  

• How does mysql define the bison YYSTYPE?
  Generally, YYSTYPE definition is generated by bison tool with directive %union. Such as, in .yy file,

  %union {
  	int ival;
  }
  

  and use it as type for nonterminal symbol,

  %type <ival> Iconst
  

  Bison tool will generate a definition in .c or .cc file,

  union YYSTYPE {
    int ival;
  };
  

  However, mysql does not use the %union directive and defines YYSTYPE type in a separate header file sql/parser_yystype.h,

  union YYSTYPE {
    Lexer_yystype lexer;  // terminal values from the lexical scanner
  };
  

  The Lexer_yystype is defined as,

  union Lexer_yystype {
    LEX_STRING lex_str;
    LEX_SYMBOL keyword;
    const CHARSET_INFO *charset;
    PT_hint_list *optimizer_hints;
    LEX_CSTRING hint_string;
  };
  

  Then, mysql defines its nonterminal symbol type as,

  %type <lexer.lex_str> TEXT_STRING_sys TEXT_STRING
  

• In general, yyparse function calls yylex function internally. How does mysql implement its lexical analyzer?
  From previous question, we know yyparse is replaced with MYSQLparse. As a consequence, yylex is named MYSQLlex. However, mysql does not use flex tool as its lexical analyzer. Instead, mysql implements its own lexical analysis process in class Lex_input_stream.