Deindex share URLs from search

[aliabd]

Adds /robots.txt to every gradio app (that disallows all) so that it doesn't get indexed by google and show up in searches. Example of indexed links.

[github-actions]

All the demos for this PR have been deployed at https://huggingface.co/spaces/gradio-pr-deploys/pr-2772-all-demos

[julien-c]

this should only apply to the apps hosted on *.gradio.app, right? On Spaces (and presumably when people self-host) we want the apps to be indexed...

[aliabd]

Yes I was worried about that too, but my understanding is this won't affect spaces since there won't be a /robots.txt route.

But it will affect the <subdomain>.hf.space and it will affect links when users self host, which is a good point. Maybe we do it in the gradio-api-server side only for share links. That will avoid spaces entirely and self hosting, so is probably the right way to go.

[aliabd]

Turns out it's non trivial to do this on the api server side so will keep it here. But I changed it to only disallow indexing when share=True which means it shouldn't affect self hosting or spaces.