net.http_proxy : Use a url when using a proxy instead of only the path

[KyleFontenot]

(Re-submitted the PR, I didn't branch my work appropriately and broke history. #25169 )

Objective:

Change the net.http_proxy.http_do method to construct a url for the request instead of just the path:
GET http://google.com/get HTTP/1.1 instead of GET /get HTTP/1.1

It is to follow the RFC 7230 standard section 5.3.2:

5.3.2. absolute-form

When making a request to a proxy, other than a CONNECT or server-wide
OPTIONS request (as detailed below), a client MUST send the target
URI in absolute-form as the request-target.

absolute-form = absolute-URI

The proxy is requested to either service that request from a valid
cache, if possible, or make the same request on the client's behalf
to either the next inbound proxy server or directly to the origin
server indicated by the request-target. Requirements on such
"forwarding" of messages are defined in Section 5.7.

An example absolute-form of request-line would be:

GET http://www.example.org/pub/WWW/TheProject.html HTTP/1.1

The change is only in http_proxy.v so it should only happen when using a proxy.

fn (pr &HttpProxy) http_do(host urllib.URL, method Method, path string, req &Request) !Response {
	host_name, port := net.split_address(host.hostname())!

	full_url := if host.scheme == 'http' || host.scheme == 'https' {
		port_part := if port == 80 || port == 0 { '' } else { ':${port}' }
		'${host.scheme}://${host_name}${port_part}${path}'
	} else {
		path
	}

	s := req.build_request_headers(req.method, host_name, port, full_url)

This change allowed a proxied http request to work for me.

My reproducing case:

proxy := http.new_http_proxy(os.environ()['HTTP_PROXY']) or { panic('failed to create proxy') }
testrequest := http.prepare(http.FetchConfig{
  url:         'http://google.com'
  proxy:       proxy
  max_retries: 2
}) or { panic('failed') }

result := testrequest.do()!
println(result.body)

@spytheman

Is there a way to do an end to end test, with a supported http proxy, instead of doing unit tests with mocks, that mimic the actual logic of the implementation, and are thus unnecessarily brittle to new changes?

That is a good question, and I thought about that during tests, but assumed a real proxy request attempt in testing would be problematic. From what I know, free public proxy servers change a lot so theyre not reliable. Having anyone reproducing the test that pings some random proxy server sounds a bit iffy. So a more internal solution makes more sense.

Only thing I can think of is if maybe vlang.io had an endpoint serving as a proxy and have it protected with an authentication key of sorts. If the user doesnt have the auth key detected in a compile-time check, skip the test. Without authentication, I'd worry about scrapers.

We could test via running a proxy server service within the test, but that would need a user's custom port open and it'd treat it as localhost which defeats a bit of the purpose. What do you think?

[huly-for-github]

Connected to Huly®: V_0.6-24668

[KyleFontenot]

I changed the test to only run if the user has a http_proxy/HTTP_PROXY/https_proxy/HTTPS_PROXY env var set. If it set, then it attempts to receive a custom header back from https://httpbin.org/headers (just echoing the header into the Response).

with this current commit in http_proxy.v on line 92:

//Before:
  s := req.build_request_headers(req.method, host_name, port, path)
//After:
  s := req.build_request_headers(req.method, host_name, port, '${host.scheme}://${host_name}${port_part}${path}')

I can confirm that the test request to httpbin.org receives a 200 status code compared to before.
But the response I get from the successful test though is from my proxy server instead of httpbin.org and it does not have the echoed custom header despite the 200 status code. Possibly might just be my specific proxy server. ?

[spytheman]

I am thinking of installing tinyproxy on one of the CI jobs, and testing through it.

[spytheman]

Excellent work.
Thank you 🙇🏻.