Add BOM for UTF-8 to improve support for non-english characters

[leonidasmi]

Context

Summary

This PR can be summarized in the following changelog entry:

• Improves support for non-english characters in llms.txt, for servers that don't serve .txt files in UTF-8.

Relevant technical choices:

Test instructions

Test instructions for the acceptance test before the PR gets merged

This PR can be acceptance tested by following these steps:

• Make sure you set your llms.txt file to have a non-english character
  • For example, add ελληνικα in your Site Title
• Generate your llms.txt file and then view it in the browser
• You'll see the # Ελληνικα: part properly and not gorbled
• Add the following snippet, that removes the BOM prefix from the llms.txt file:

add_filter( 'wpseo_llmstxt_encoding_prefix', 'custom_llmstxt_encoding_prefix', 10 );
function custom_llmstxt_encoding_prefix() {
    return '';
}

• Re-generate the file and view it in the browser again
• This time, you'll see that part as garbled: # Ελληνικα:

Relevant test scenarios

• Changes should be tested with the browser console open
• Changes should be tested on different posts/pages/taxonomies/custom post types/custom taxonomies
• Changes should be tested on different editors (Default Block/Gutenberg/Classic/Elementor/other)
• Changes should be tested on different browsers
• Changes should be tested on multisite

Test instructions for QA when the code is in the RC

• QA should use the same steps as above.

Impact check

This PR affects the following parts of the plugin, which may require extra testing:

• Check with this RC and the released versions, that you get the same content if there are no non-english characters in the llms.txt file.

UI changes

• This PR changes the UI in the plugin. I have added the 'UI change' label to this PR.

Other environments

• This PR also affects Shopify. I have added a changelog entry starting with [shopify-seo], added test instructions for Shopify and attached the Shopify label to this PR.

Documentation

• I have written documentation for this change. For example, comments in the Relevant technical choices, comments in the code, documentation on Confluence / shared Google Drive / Yoast developer portal, or other.

Quality assurance

• I have tested this code to the best of my abilities.
• During testing, I had activated all plugins that Yoast SEO provides integrations for.
• I have added unit tests to verify the code works as intended.
• If any part of the code is behind a feature flag, my test instructions also cover cases where the feature flag is switched off.
• I have written this PR in accordance with my team's definition of done.
• I have checked that the base branch is correctly set.

Innovation

• No innovation project is applicable for this PR.
• This PR falls under an innovation project. I have attached the innovation label.
• I have added my hours to the WBSO document.

Fixes #

[coveralls]

Pull Request Test Coverage Report for Build b158b26bd4504ab7f9d7f7222a4a81a3654ed313

Details

• 4 of 4 (100.0%) changed or added relevant lines in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.003%) to 53.158%

---

Totals
Change from base Build 277ef9ccecf3d7c24d6e10a2b5501997e9f2d89b:	0.003%
Covered Lines:	29826
Relevant Lines:	57038

---

💛 - Coveralls

[thijsoo]

CR + ACC 👍