Python yarl (aiohttp) breaks URLs

The python aiohttp library uses yarl for URLs internally, and yarl normalizes URLs by default. It silently decodes some %-encoded characters in the query string that do not strictly need to be encoded.

Sounds harmless, but it isn't. Changing the URL breaks any protocol that signs important aspects of a HTTP request for security.

Took me a while to find this bug. I usually expect an HTTP client library to not silently manipulate the URLs before sending a request. Smarter is not always better

Yes, this is documented behavior, but I rarely read the documentation for HTTP client libraries. I know how HTTP works, I know how those libraries should work, and silently altering URLs without any need is something I really did not expect from a low level HTTP lib.

Imagine curl would silently change the order of headers or query parameters or remove an explicitly defined content-length header because it decides that chunked transfer is better for you. Stuff would break.


Imported from: https://chaos.social/@defnull/115488120527803714