Deal with pages that have been selected and then got deleted/private/noindexed etc.

[leonidasmi]

Context

Summary

This PR can be summarized in the following changelog entry:

• Deals with pages that have been selected and then got deleted/private/noindexed etc.

Relevant technical choices:

• Reverts the registering of the policy pages (via the get_site_basics_policies() function) back to what it was before the feature branch). Mainly because we now divert enough from that way for the llms.txt pages so we can reduce the impact of our changes now.

Test instructions

Test instructions for the acceptance test before the PR gets merged

This PR can be acceptance tested by following these steps:

• Select a couple of pages to be added manually in the llms.txt file
• After saving the settings, do the following to those saved pages (make sure at least one of them are from the contact/about/term/etc pages and one of them from the content pages):
  • Make one private
  • Make one draft
  • Make one scheduled
  • Make one noindexed
  • Make one password-protected
  • Delete one
• Refresh the llms.txt settings page and confirm that you don't see any of those pages as selected. Instead, the respective dropdowns are empty and ready to be selected again
  • Make sure to check both when the \available_pages request is not done and after it's done. Should be empty in both
• Dont save and revert one of those pages to the public state it was before
• Refresh the settings page and confirm that the reverted page re-appeared in the settings it was before.
• Make it again private/trashed/etc
• Go to the settings page again and this time, add another page at the end of the selections and save.
• Refresh the settings page and confirm that you see the selections you would expect.
• Now run the scheduled action and confirm that you dont see any pages that are private/draft/etc
  • This should have also cleaned up the IDs of the private pages from the db In a later PR
  • To check that, check the wpseo_llmstxt option in the db and confirm that you dont see those IDs in the about_us_page, contact_page, terms_page, privacy_policy_page, shop_page, and other_included_pages attributes. In a later PR
• Repeat the above tests, but this time with indexables reset and disabled
  • The only difference would be that the noindexed pages will be treated as public ones.

Deal with pages with no titles:

• Create a page with no title
• Go to the llms.txt settings and when clicking the dropdown, it should appear as its slug (eg. 123)
• Select it and save
• Refresh the page and confirm that the dropdown is populated even before the /available_pages request is completed
  • you might need to add the first filter from this PR, to make sure you delay that request enough for you to check)

Relevant test scenarios

• Changes should be tested with the browser console open
• Changes should be tested on different posts/pages/taxonomies/custom post types/custom taxonomies
• Changes should be tested on different editors (Default Block/Gutenberg/Classic/Elementor/other)
• Changes should be tested on different browsers
• Changes should be tested on multisite

Test instructions for QA when the code is in the RC

• QA should use the same steps as above.

Impact check

This PR affects the following parts of the plugin, which may require extra testing:

UI changes

• This PR changes the UI in the plugin. I have added the 'UI change' label to this PR.

Other environments

• This PR also affects Shopify. I have added a changelog entry starting with [shopify-seo], added test instructions for Shopify and attached the Shopify label to this PR.

Documentation

• I have written documentation for this change. For example, comments in the Relevant technical choices, comments in the code, documentation on Confluence / shared Google Drive / Yoast developer portal, or other.

Quality assurance

• I have tested this code to the best of my abilities.
• During testing, I had activated all plugins that Yoast SEO provides integrations for.
• I have added unit tests to verify the code works as intended.
• If any part of the code is behind a feature flag, my test instructions also cover cases where the feature flag is switched off.
• I have written this PR in accordance with my team's definition of done.
• I have checked that the base branch is correctly set.

Innovation

• No innovation project is applicable for this PR.
• This PR falls under an innovation project. I have attached the innovation label.
• I have added my hours to the WBSO document.

Fixes #

[coveralls]

Pull Request Test Coverage Report for Build 89b1dc57ed7d5814114d7b687159e0c5e1855337

Details

• 1 of 38 (2.63%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.004%) to 53.095%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/llms-txt/infrastructure/content/manual-post-collection.php	0	7	0.0%
src/integrations/settings-integration.php	1	31	3.23%

Totals
Change from base Build b9d37ade652cfe5bc879128c0470f2f840e98818:	-0.004%
Covered Lines:	29985
Relevant Lines:	57354

---

💛 - Coveralls

[thijsoo]

CR + ACC 👍