Input format - The Rust Reference

## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

[input]

# Input format

[input.syntax]

Lexer
CHAR → <a Unicode scalar value>

NUL → U+0000

[input.intro]

This chapter describes how a source file is interpreted as a sequence of tokens.

See Crates and source files for a description of how programs are organised into files.

[input.encoding]

## Source encoding

[input.encoding.utf8]
Tests
Tests with this rule:

- tests/ui/macros/not-utf8.rs
- tests/ui/parser/utf16-be-without-bom.rs
- tests/ui/parser/utf16-le-without-bom.rs

Each source file is interpreted as a sequence of Unicode characters encoded in UTF-8.

[input.encoding.invalid]
Tests
Tests with this rule:

- tests/ui/macros/not-utf8.rs

It is an error if the file is not valid UTF-8.

[input.byte-order-mark]
Tests
Tests with this rule:

- tests/ui/codemap_tests/utf8-bom.rs
- tests/ui/json/json-bom-plus-crlf-multifile.rs
- tests/ui/json/json-bom-plus-crlf.rs

## Byte order mark removal

If the first character in the sequence is `U+FEFF` (BYTE ORDER MARK), it is removed.

[input.crlf]
Tests
Tests with this rule:

- tests/ui/json/json-bom-plus-crlf-multifile.rs
- tests/ui/json/json-bom-plus-crlf.rs
- tests/ui/lexer/lexer-crlf-line-endings-string-literal-doc-comment.rs

## CRLF normalization

Each pair of characters `U+000D` (CR) immediately followed by `U+000A` (LF) is replaced by a single `U+000A` (LF). This happens once, not repeatedly, so after the normalization, there can still exist `U+000D` (CR) immediately followed by `U+000A` (LF) in the input (e.g. if the raw input contained “CR CR LF LF”).

Other occurrences of the character `U+000D` (CR) are left in place (they are treated as whitespace).

[input.shebang]
Tests
Tests with this rule:

- tests/ui/parser/shebang/issue-71471-ignore-tidy.rs
- tests/ui/parser/shebang/shebang-comment.rs
- tests/ui/parser/shebang/shebang-empty.rs
- tests/ui/parser/shebang/shebang-must-start-file.rs
- tests/ui/parser/shebang/shebang-space.rs
- tests/ui/parser/shebang/valid-shebang.rs

## Shebang removal

[input.shebang.intro]

If the remaining sequence begins with the characters `#!`, the characters up to and including the first `U+000A` (LF) are removed from the sequence.

For example, the first line of the following file would be ignored:

    #!/usr/bin/env rustx

    fn main() {
        println!("Hello!");
    }

[input.shebang.inner-attribute]
Tests
Tests with this rule:

- tests/ui/parser/shebang/multiline-attrib.rs
- tests/ui/parser/shebang/regular-attrib.rs
- tests/ui/parser/shebang/shebang-and-attrib.rs
- tests/ui/parser/shebang/shebang-doc-comment.rs
- tests/ui/parser/shebang/sneaky-attrib.rs

As an exception, if the `#!` characters are followed (ignoring intervening comments or whitespace) by a `[` token, nothing is removed. This prevents an inner attribute at the start of a source file being removed.

[input.tokenization]

## Tokenization

The resulting sequence of characters is then converted into tokens as described in the remainder of this chapter.

> Note

> The standard library `include!` macro applies the following transformations to the file it reads:

> - Byte order mark removal.
> - CRLF normalization.
> - Shebang removal when invoked in an item context (as opposed to expression or statement contexts).

> The `include_str!` and `include_bytes!` macros do not apply these transformations.