Merge pull request #1427 from Open-Source-Legal/dependabot/pip/tokenizers-gte-0.23.1-and-lt-0.24

Update tokenizers requirement from <0.23,>=0.21 to >=0.23.1,<0.24

Author: JSv4

requirements/base.txt

@@ -12,7 +12,7 @@ pydantic==2.*
 typing-extensions==4.*  # https://github.com/python/typing_extensions
 requests>=2.33.1,<3  # https://requests.readthedocs.io/en/latest/
 httpx>=0.28.1,<1  # https://github.com/encode/httpx - async HTTP for agent tools
-tokenizers>=0.21,<0.23  # Pin to prevent conflicts with transformers
+tokenizers>=0.23.1,<0.24  # Pin to prevent conflicts with transformers
 
 
 # Django