Tokens¶

During lexical analysis, source file text is separated into distinctive meaningful tokens.

Each source contains at least one end-of-file token.

Identifiers¶

The primary purpose of identifiers is to give a human-readable name to a specific program part.

identifier = identifier_or_keyword | escaped_identifier ;

Regular identifier¶

A regular identifier begins with a letter or underscore and is followed by any number of letters, digits, underscores, or marks.

identifier_or_keyword = identifier_start { identifier_continue } ;

identifier_start      = "_" | letter ;
identifier_continue   = "_" | letter | digit ;

letter = <Lu> | <Ll> | <Lt> | <Lm> | <Lo> | <Nl> ; (* (1)! *)
digit  = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ;

1. Wikipedia page about Unicode categories: https://en.wikipedia.org/wiki/Unicode_character_property#General_Category

Escaped identifier¶

An escaped identifier is enclosed in backtick characters and may contain any character, including those that are otherwise invalid in identifiers. Escape sequences are supported inside escaped identifiers.

escaped_identifier
    = "`" { escaped_identifier_character | identifier_escape_sequence } "`" ;

escaped_identifier_character
    = ? any Unicode scalar value except LF, CR, "\", and "`" ? ;

identifier_escape_sequence
    = "\\"
    | "\`"
    | "\a"
    | "\b"
    | "\f"
    | "\n"
    | "\r"
    | "\t"
    | "\v"
    | "\0"
    | "\u" hex_digit , hex_digit , hex_digit , hex_digit
    | "\U" hex_digit , hex_digit , hex_digit , hex_digit ,
           hex_digit , hex_digit , hex_digit , hex_digit
    ;

hex_digit
    = digit
    | "A" | "B" | "C" | "D" | "E" | "F"
    | "a" | "b" | "c" | "d" | "e" | "f"
    ;

Punctuators and operators¶

Literals¶

Literals are tokens that are used in literal expressions.

Numeric literals¶

numeric_literal = ( float_literal | integer_literal ) , [ numeric_suffix ] ;

integer_literal = decimal_literal | hex_literal | binary_literal ;

decimal_literal = digit , { { separator } , digit } ;
binary_literal  = "0b" , { { separator } , binary_digit } ;
hex_literal     = "0x" , { { separator } , hex_digit } ;

float_literal  = decimal_literal , "." , decimal_literal
               | decimal_literal , [ "." , decimal_literal ] , float_exponent;

float_exponent = ( "e" | "E" ) , [ "+" | "-" ] , decimal_literal;

separator    = "'" ;
binary_digit = "0" | "1" ;
digit        = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ;
hex_digit    = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
             | "A" | "B" | "C" | "D" | "E" | "F"
             | "a" | "b" | "c" | "d" | "e" | "f"
             ;

numeric_suffix
    = "u8" | "u16" | "u32" | "u64" | "u128" | "usize"
    | "i8" | "i16" | "i32" | "i64" | "i128" | "isize"
    | "f16" | "f32" | "f64"
    ;

Character and string literals¶

string_literal    = '"' { character - '"' } '"' [ encoding_suffix ] ;
character_literal = "'" ( character - "'" ) "'" [ encoding_suffix ] ;

character = ? any Unicode scalar value except LF, CR, "\" ? | escape_sequence;

escape_sequence
    = "\\"
    | "\'"
    | '\"'
    | "\n"
    | "\r"
    | "\t"
    | "\0"
    | "\u" hex_digit , hex_digit , hex_digit , hex_digit
    | "\U" hex_digit , hex_digit , hex_digit , hex_digit ,
           hex_digit , hex_digit , hex_digit , hex_digit
    ;

hex_digit
    = digit
    | "A" | "B" | "C" | "D" | "E" | "F"
    | "a" | "b" | "c" | "d" | "e" | "f"
    ;

encoding_suffix = "ascii" | "utf8" | "utf16" | "utf32" ;

Character escape sequences¶

An escape starts with a Backslash (U+005C) and continues with one of the following forms:

• The backslash escape is the Backslash (U+005C) which must be escaped in order to denote itself.
• The double quote escape is the Double Quote (U+0022). This escape is optional for literals that are not enclosed by double quote characters.
• The apostrophe (a.k.a. single quote) escape is the Apostrophe (U+0027). This escape is optional for literals that are not enclosed by apostrophe characters.
• A whitespace escape is one of the characters:
  • Lower n (U+006E) denoting the Line Feed (U+000A).
  • Lower r (U+0072) denoting the Carriage Return (U+000D).
  • Lower t (U+0074) denoting the Horizontal Tab (U+0009).
• The null escape is the character Zero (U+0030) and denotes the Unicode Null (U+0000) value.
• The any-Unicode-scalar-value escape is the Lower u (U+0075) followed by four hexadecimal digits, or the Capital U (U+0055) followed by eight hexadecimal digits.

Keywords¶

Keywords are much like identifiers, except they cannot be used as such. They are reserved words that the language uses in different contexts.

Strong keywords¶

Strong keywords are a set of words that are always treated as non-identifier tokens. The following list represents all existing strong keywords in the current language version.

• i8
• i16
• i32
• i64
• i128
• isize
• u8
• u16
• u32
• u64
• u128
• usize
• bool
• true
• false
• module
• use
• if
• else
• for
• do
• while
• loop
• continue
• break

Contextual keywords¶

Contextual keywords are identifiers that have a special meaning only in certain syntactic contexts. Outside of these contexts, they can be used as regular identifiers and as names for user-defined program parts.

The following list represents all contextual keywords in the current language version.

Reserved keywords¶

Reserved keywords are identifiers that are reserved for future use in the language. Using a reserved keyword as an identifier will result in a compilation error.

• enum ^{maybe add a link to the discussion?}
• as
• auto
• null
• interface
• virt
• const