Pexels Photos Downloader

Introduction

In this post I shall demonstrate how I wrote a HTTP client application in Go to download photos from Pexels.

Pexels provides a platform which allows you to download stock photos and videos for free.

I chose Go as the language to write in since I wanted to learn how to use it.

API

In order to use the API you will need:

  1. Account
  2. API Key

Documentation: RESTful API

 Restrictions

  • Rate-limited to 200 requests per hour
  • 20,000 requests per month

API URLs

URL Description
https://api.pexels.com/v1 Photos base URL
https://api.pexels.com/videos Videos base URL

Authentication

The API Key is sent in the HTTP Authorization request header.

curl -H "Authorization: <API_KEY>" "https://api.pexels.com/v1/search?query=<QUERY>"

Response example

Truncated Response

{
   "total_results":10000,
   "page":1,
   "per_page":15,
   "photos":[
      {
         "id":3573351,
         "width":3066,
         "height":3968,
         "url":"https://www.pexels.com/photo/trees-during-day-3573351/",
         "photographer":"Lukas Rodriguez",
         "photographer_url":"https://www.pexels.com/@lukas-rodriguez-1845331",
         "photographer_id":1845331,
         "src":{
            "original":"https://images.pexels.com/photos/3573351/pexels-photo-3573351.png",
            "large2x":"https://images.pexels.com/photos/3573351/pexels-photo-3573351.png?auto=compress&cs=tinysrgb&dpr=2&h=650&w=940",
            "large":"https://images.pexels.com/photos/3573351/pexels-photo-3573351.png?auto=compress&cs=tinysrgb&h=650&w=940",
            "medium":"https://images.pexels.com/photos/3573351/pexels-photo-3573351.png?auto=compress&cs=tinysrgb&h=350",
            "small":"https://images.pexels.com/photos/3573351/pexels-photo-3573351.png?auto=compress&cs=tinysrgb&h=130",
            "portrait":"https://images.pexels.com/photos/3573351/pexels-photo-3573351.png?auto=compress&cs=tinysrgb&fit=crop&h=1200&w=800",
            "landscape":"https://images.pexels.com/photos/3573351/pexels-photo-3573351.png?auto=compress&cs=tinysrgb&fit=crop&h=627&w=1200",
            "tiny":"https://images.pexels.com/photos/3573351/pexels-photo-3573351.png?auto=compress&cs=tinysrgb&dpr=1&fit=crop&h=200&w=280"
         },
         "liked":false
      }
   ],
   "next_page":"https://api.pexels.com/v1/search/?page=2&per_page=15&query=nature"
}

As illustrated above the JSON response contains an array of photos.

Photo

A Pexels photo can be described as having:

Property Type
id int
width int
height int
url string
photographer string
photographer_url string
photographer_id int
src object

Client Application

Go type definitions

In the application I defined a new type called pexelPhotoResponse

This type encapsulated the photo object taken from the JSON response.

type pexelPhotoResponse struct {
	URL          string
	TotalResults int `json:"total_results"`
	Page         int `json:"page"`
	PerPage      int `json:"per_page"`
	Photos       []struct {
		ID              int    `json:"id"`
		Width           int    `json:"width"`
		Height          int    `json:"height"`
		URL             string `json:"url"`
		Photographer    string `json:"photographer"`
		PhotographerURL string `json:"photographer_url"`
		PhotographerID  int    `json:"photographer_id"`
		Src             struct {
			Original  string `json:"original"`
			Large2X   string `json:"large2x"`
			Large     string `json:"large"`
			Medium    string `json:"medium"`
			Small     string `json:"small"`
			Portrait  string `json:"portrait"`
			Landscape string `json:"landscape"`
			Tiny      string `json:"tiny"`
		} `json:"src"`
		Liked bool `json:"liked"`
	} `json:"photos"`
	NextPage string `json:"next_page"`

Another type created called args.

This type encapsulated the command line arguments provided by a user.

type args struct {
	APIKey    string `API Key`
	Query     string `Search Query`
	Photos    int    `Number of Photos to request`
	PhotoSize string `Photo size`
	Output    string `Path to store Photos`
}

Creating a HTTP request

In the example below a request to the API is created.

Two parameters are defined.

  1. A sending channel as as a parameter response chan<- pexelPhotoResponse
  2. A pointer to the command line arguments requestArgs args

The rest of the code consumes the response body, in this case this is the JSON encoded response.

json.Unmarshal(body, &responseBody)

Parses the JSON-encoded data and stores the result in the value pointed to by responseBody

responseBody points to the address of an object of type pexelPhotoResponse - which is the struct type described earlier.


//PexelsRequest sender makes request to the API
func PexelsRequest(response chan<- pexelPhotoResponse, requestArgs *args) {
	log.Println("Making request for photos.")

	client := httpClient()

	req, err := http.NewRequest("GET", pexelAPI+"/search?query="+requestArgs.Query+"&locale="+defaultLocale+"&per_page="+strconv.Itoa(perPage), nil)

	req.Header.Add("Authorization", requestArgs.APIKey)

	if err != nil {
		log.Fatal("Request failed ", err)
	}

	resp, err := client.Do(req)

	if resp.StatusCode != 200 {
		body, _ := ioutil.ReadAll(resp.Body)
		fmt.Println(string(body))
		log.Fatal("Request status = ", resp.StatusCode)

	} else {
		log.Println("Response status = ", strconv.Itoa(resp.StatusCode)+" (OK)")
	}

	defer resp.Body.Close()

	var responseBody pexelPhotoResponse

	body, err := ioutil.ReadAll(resp.Body)

	if err != nil {
		log.Fatal("Error reading body. ", err)
	}

	json.Unmarshal(body, &responseBody)
	response <- responseBody
	defer wg.Done()
}

Parsing the JSON response

The function below uses a receiver only channel input <-chan pexelPhotoResponse

  • A loop iterates through the array of photos.

  • As the user has to input the size of the photo they want to download, the response.Photos[i].Src is used - imageSrc is then set depending on which size the user had entered

  • The createFile function creates a file using the unique photo id and appending a ‘.png’ file extension since all the photos are in the same format.

//PexelsResponse processes the response from the API
func PexlesResponse(input <-chan pexelPhotoResponse, requestArgs *args) {
	wg.Add(1)

	response := <-input
	if len(response.Photos) > 0 {

		for i := 0; i < response.PerPage; i++ {
			imageSrc := ""
			switch requestArgs.PhotoSize {
			case "original":
				imageSrc = response.Photos[i].Src.Original
			case "large2x":
				imageSrc = response.Photos[i].Src.Large2X
			case "large":
				imageSrc = response.Photos[i].Src.Large
			case "medium":
				imageSrc = response.Photos[i].Src.Medium
			case "small":
				imageSrc = response.Photos[i].Src.Small
			case "portrait":
				imageSrc = response.Photos[i].Src.Portrait
			case "landscape":
				imageSrc = response.Photos[i].Src.Landscape
			case "tiny":
				imageSrc = response.Photos[i].Src.Tiny
			default:
				panic("invalid image size")
			}
			file := createFile(requestArgs.Output + "/" + strconv.Itoa(response.Photos[i].ID) + ".png")
			saveFile(file, httpClient(), imageSrc)

			if saveCount == requestArgs.Photos {
				absPath, _ := filepath.Abs(requestArgs.Output)
				getRequestStats(requestArgs.APIKey, httpClient())
				log.Println("Photos saved in " + absPath)
				os.Exit(0)
			}
		}
		if response.NextPage != "" {
			wg.Add(1)
			PexelsRequest(res, requestArgs, response.NextPage)
			PexlesResponse(res, requestArgs)

			//wg.Done()
		}
	} else {
		log.Println("0 results returned.")
	}

	defer wg.Done()
}

Downloading files

This function takes 3 parameters

  1. Pointer to a file object //this will be the file to save to
  2. Pointer to a http client object //this will be used to make the http request for the photo
  3. A string //this holds the photo src URL to download the photo from
func saveFile(file *os.File, client *http.Client, fullURLFile string) {
	resp, err := client.Get(fullURLFile)

	checkError(err)

	defer resp.Body.Close()

	_, err = io.Copy(file, resp.Body)

	defer file.Close()

	checkError(err)

	base := filepath.Base(file.Name())

	checkError(err)
	saveCount++

	log.Println("File " + strconv.Itoa(saveCount) + ", saved as: " + base)
}

Main

Entrypoint to application.

func main() {

      // define command line argument flags
	key := flag.String("key", "", "Your API key")
	query := flag.String("query", "", "Search term")
	photos := flag.Int("photos", 100, "Max number of photos to download.")
	photoSize := flag.String("size", "", "Size of the photo to download \n(original,large2x,large,medium,small,portrait,landscape,tiny)\n")
	output := flag.String("output", "", "Path to where to store the images")
	flag.Parse()

	var inputArgs args

	inputArgs.APIKey = *key
	inputArgs.Query = *query
	inputArgs.Photos = *photos
	inputArgs.PhotoSize = *photoSize
	inputArgs.Output = *output


      // argument validation
	checkVars(&inputArgs)

      // create output directory but only if it doesn't exist
	createDir(inputArgs.Output)

      // add a weight group before we call the goroutines
      wg.Add(1)


	go PexelsRequest(res, &inputArgs, pexelAPI+"/search?query="+inputArgs.Query+"&locale="+defaultLocale+"&per_page="+strconv.Itoa(perPage))
	go PexlesResponse(res, &inputArgs)

      // block until the goroutine above are complete
      wg.Wait()
}

Goroutines

Both PexlesResponse() and PexelsRequest() functions are called as goroutrines from the main function. Since they are running asynchronously, in separate goroutines they needs to be a way to wait from them to finish.

Waitgroups

To wait for multiple goroutines to finish wait groups were added to the code.

You may have noticed the use of wg.Done() and wg.Add()

wg.Add()

  • Add adds delta, which may be negative, to the WaitGroup counter.

  • If the counter becomes zero, all goroutines blocked on Wait are released. If the counter goes negative, Add panics.

wg.Done()

  • Done decrements the WaitGroup counter by one.

wg.Wait()

  • Block until the WaitGroup counter goes back to 0; all the workers notified they’re done.

Usage

Running the application without any arguments prints the usage.

go run pexels.go
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Photos provided by Pexels.
API Restrictions = Rate-limited to 200 requests per hour and 20,000 requests per month.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020/06/18 17:30:05 Checking supplied arguments.
Usage of pexels:
  -key string
        Your API key
  -output string
        Path to where to store the photos
  -photos int
        Max number of photos to download. (default 100)
  -query string
        Search term
  -size string
        Size of the photo to download
        (original,large2x,large,medium,small,portrait,landscape,tiny)

2020/06/18 17:30:05 Invalid arguments
exit status 1

Example.

This will:

  • query the API for ’landscapes’ photos

  • select an original size to download

  • only download 6 photos to disk

  • store the downloaded photos into a local directory called ’landscapes’

  • prints the requests remaining (20,0000 requests are available per month)

go run pexels.go -key=<API_KEY> -query=landscape -size=original -photos=6 -output=/var/tmp/landscapes

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Photos provided by Pexels.
API Restrictions = Rate-limited to 200 requests per hour and 20,000 requests per month.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020/06/18 17:49:30 Checking supplied arguments.
2020/06/18 17:49:31 File 1, saved as: 414171.png
2020/06/18 17:49:32 File 2, saved as: 132037.png
2020/06/18 17:49:33 File 3, saved as: 814499.png
2020/06/18 17:49:33 File 4, saved as: 747964.png
2020/06/18 17:49:33 File 5, saved as: 917494.png
2020/06/18 17:49:33 File 6, saved as: 36717.png
2020/06/18 17:49:34 Requests remaining 1330914
2020/06/18 17:49:34 Photos saved in /var/tmp/landscapes


----

ls -lrt /var/tmp/landscapes

 414171.png
-rw-r--r-- 1  5.7M Jun 18 17:49 132037.png
-rw-r--r-- 1  13M Jun 18 17:49 814499.png
-rw-r--r-- 1  6.1M Jun 18 17:49 747964.png
-rw-r--r-- 1  1.4M Jun 18 17:49 917494.png
-rw-r--r-- 1  182K Jun 18 17:49 36717.png

Further reading

Last updated on 18 Jun 2020
Published on 18 Jun 2020