Golang – Random User-Agent’s & Proxies

When scraping the internet it often makes sense to rotate both the proxy and user agent sent along with the HTTP request. By rotating proxies and user agents we can decrease detection, and avoid having our IP addresses banned, or face being rate limited by the site in question. This is relatively easy to do in Golang, though it is not quite as simple as in Python.

Generating A Random Choice

First we need to write a small function which takes a slice of string and returns a string. We first seed Go’s random generating with the computer’s current Unix time. Strictly speaking this will make our results only pseudo-random, as it would be possible to produce the same result given the same seed. This doesn’t matter as pseudo-randomness is good enough for the purposes of web scraping.

Creating A Client

Making a HTTP request in Go requires that we create a client to make the request with. The above function takes in the URL we are looking to scrape, our slice of proxies and our slice of User-Agent strings. We then use our RandomString function to pick both our proxy and our User-Agent. The parsed proxy string is then passed into our client by creating a custom transport using the said proxy. We then create a request to be executed by the client and add our custom user agent header to the request. We then use the client to make the specified request and return the HTTP response should everything go fine.

Example Usage

We can then use the function as above. By simply creating a slice of strings to hold both our User-Agents and our proxies, we can then pass these slices straight to the function. In the example, we don’t do anything interesting with the response but you will be able to manipulate the http.response anyway you want. The full code usded can be found here.

Leave a Reply

Your email address will not be published. Required fields are marked *