Skip to content

build error from tokenizers dependency #340

Open

Description

@joprice

I'm seeing the same rust build error reported here huggingface/tokenizers#1485, which was fixed in huggingface/tokenizers#1355 (tokenizers-0.14.1). Running pip install --dry-run transformers==4.34.0 shows that the version is the one including the fix. Is it possible to update transformers to a more recent version?

Otherwise, there is this workaround suggested: RUSTUP_TOOLCHAIN=1.72.0 pip install tokenizers==0.13.2, but it another error

  error: package `clap v4.5.26` cannot be built because it requires rustc 1.74 or newer, while the currently active rustc version is 1.72.0```.

Activity

  1. joprice commented on Jan 18, 2025

    @joprice
    Author

    I was able to build by setting RUSTFLAGS="-A invalid_reference_casting" to ignore the warning, but seems like since it can lead to UB, it should be addressed eventually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions