Don't use robots.txt for Twitter/X

As of a few months ago, Twitter/X.com's robots.txt policy prevents the Discord client from creating embeds/previews of Twitter content. The Discord client's decision to respect that policy makes Discord a very inconvenient platform for sharing information from Twitter.

Suggestion: Do not respect Twitter's robots.txt policy, and instead retrieve any content you wish. Robots.txt is a voluntary standard and not one clients are in any way obligated to respect.

Further reading: https://en.wikipedia.org/wiki/Robots.txt