Querying Last.fm web API with F# // Bohdan Stupak's blog

Introduction

Let’s imagine that you have an edgy musical taste so you would like to recommend to your friends only those artists which are the most mainstream. If you have a profile on last.fm, then you could write a small tool which would query and process your listening statistics to automate this task.

My tool for this job is F# programming language and I’ll show you some benefits of it such as type providers or the ease of unit testing functions comparing to objects.

The complete source code can be accessed here.

Workflow

The task is as follows:

Get top 50 artists from last.fm chart.
Transform them to the collection of their names.
Remove those ones which were already recommended.
Replace non-url characters in artist names to make further API call.
For each artist, make additional API call to get a number of listeners.
Transform given information to a more minimalistic data type which would contain only artist name and listeners count.
Order artists by the count of listeners, implying that those which have more listeners are more mainstream.

Note how this workflow is expressed by F# pipeline in a truly idiomatic way.

let result = getTopArtists
                |> getTopArtistNames
                |> removeAlreadyRecomendedArtists
                |> getUrlEncodedArtistNames
                |> mapArtistNamesToArtistInfo getArtistInfo
                |> getArtistsShortInfo
                |> orderArtistsByListenersCount

Making Use of Type Providers

Type providers are arguably the most advertised feature of F#. Type providers allow us to access a lot of contexts such as Web APIs, database schemas, etc. as strongly typed entities which allow us to get compiler time support and some nice perks as IDE autocomplete.

To make use of it in our application, we:

Import FSharp.Data

Declare snippet of our API response:

let [<Literal>] TopArtistsSample = """{
   "topartists":{
      "artist":[
         {
            "name":"Porcupine Tree",
            //skipped for the sake of breivety
         }
      ],
      "@attr":{
         "user":"Morbid_soul",
         "page":"1",
         "perPage":"2",
         "totalPages":"165",
         "total":"330"
      }
   }
    }"""

Construct type from our sample via JsonProvider:

type TopArtists = JsonProvider<TopArtistsSample>

Enjoy compile time support of strongly typed response.

Using Higher Order Functions to Improve Unit Testing

Let’s take a close look at the following function:

let mapArtistNamesToArtistInfo getArtistInfoFn artists =
    artists
        |> Array.map (fun i -> getArtistInfoFn i)

getArtistInfoFn responds for interaction with remote web API. Here’s how the unit testing of such a scenario is performed.

let getArtistInfoStub input =
        match input with
        | "Nokturanl Mortum" -> 1
        | "Krobak" -> 2
        | _ -> 3

[<Fact>]
let mapArtistNamesToArtistInfo_returns_expected_result() =
    let result = mapArtistNamesToArtistInfo getArtistInfoStub
                   [| "Nokturanl Mortum"; "Heinali"; "Krobak"|]
    Assert.Equal(result.[0], 1)
    Assert.Equal(result.[1], 3)
    Assert.Equal(result.[2], 2)

This is far more elegant than the typical testable OO-solution which would require introducing an interface, injecting it into caller class and introducing some heavyweight mocking library inside the test project. One may argue that injecting an impure function into pure is not truly functional way but F# is quite a forgiving language and allows us not to come with some clever concepts as free monads, etc.

Error Handling

The attentive reader may have noticed that we rely on web API working faultlessly which is not a sign of robust programming. For proper handling, we will employ the concept of railway oriented programming. The main idea is to encode successful and unsuccessful execution of function into return type so that all functions in a pipeline would handle successful result with some useful business logic and the unsuccessful result would be excluded from further execution.

But I strongly encourage you not to take my word for it but read the original article instead which explains this concept in far more detail.

The recipe is the following:

type Result<'TSuccess,'TFailure> =
    | Success of 'TSuccess
    | Failure of 'TFailure

let switch switchFunction1 switchFunction2 input =
    match switchFunction1 input with
    | Success s -> switchFunction2 s
    | Failure f -> Failure f

let (>=>) switchFunction1 switchFunction2 input =
    switch switchFunction1 switchFunction2 input

Now we can wrap our return value into the provided type:

let getTopArtists () =
    try
        let path = String.Format(getTopArtistsPattern, baseUrl, userName, apiKey)
        let data = Http.Request(path)
        match data.Body with
        | Text text -> Success(TopArtists.Parse(text).Topartists.Artist)
        | _ -> Failure "getTopArtists. Unexpected format of reponse message"
    with
    | ex -> Failure ex.Message

So with this, the pipeline would transform to:

let pipeline =
    getTopArtists
        >=> getTopArtistNames
        >=> removeAlreadyRecomendedArtists
        >=> getUrlEncodedArtistNames
        >=> mapArtistNamesToArtistInfo getArtistInfo
        >=> getArtistsShortInfo
        >=> orderArtistsByListenersCount

Let’s also take a look at unit test snippet to get an overall feeling how the caller works with function output:

[<Fact>]
let orderArtistsByListenersCount_returns_expected_result() =
    let Satie = {name = "Erik Satie"; listeners = 750000}
    let Chopin = {name ="Frederic Chopin"; listeners = 1200000}
    let Barber = {name = "Samuel Barber"; listeners = 371000}
    let artists = [|Satie; Chopin; Barber|]
    let result = orderArtistsByListenersCount artists
    match result with
    | Success s ->
        Assert.Equal(s.[0], Chopin)
        Assert.Equal(s.[1], Satie)
        Assert.Equal(s.[2], Barber)
    | Failure _ -> Assert.True(false)

Using Built-in Result type

F# comes with a built-in Result type which allows us to ditch ROPHelper.

Our pipeline now looks as follows:

let pipeline =
    getTopArtists()
        |> Result.bind getTopArtistNames
        |> Result.bind removeAlreadyRecomendedArtists
        |> Result.bind getUrlEncodedArtistNames
        |> Result.bind (mapArtistNamesToArtistInfo getArtistInfo)
        |> Result.bind getArtistsShortInfo
        |> Result.bind orderArtistsByListenersCount

Note that we had to make our getTopArtists() to accept unit to be accepted by Result.bind. We pattern match the result as below:

[<Fact>]
let getUrlEncodedArtistNames_returns_expected_result() =
    let result = getUrlEncodedArtistNames [|"Bohren & Der Club Of Gore"; "Цукор Біла Смерть"|]
    match result with
    | Ok s ->
        Assert.Equal(s.[0], "Bohren+%26+Der+Club+Of+Gore")
        Assert.Equal(s.[1],
        "%d0%a6%d1%83%d0%ba%d0%be%d1%80+%d0%91%d1%96%d0%bb%d0%b0+%d0%a1%d0%bc%d0%b5%d1%80%d1%82%d1%8c")
    | Error _ -> Assert.True(false)

Property-based testing

False safety of traditional unit-testing approach

Let’s imagine that we have an array of musicians where we define a musician by his/her name and an overall number of people who have listened to that musician:

type Artist = {
    name: string
    listeners: int
}

And also, we have a function that orders a descendant array of type above by listeners count.

let orderArtistsByListenersCount artists =
    let ordered =
            artists
                |> Array.sortBy (fun i -> -i.listeners)
    Success(ordered)

We could write a unit-test to verify that this function works correctly with xUnit just as below.

[<Fact>]
let orderArtistsByListenersCount_returns_expected_result() =
    let Satie = {name = "Erik Satie"; listeners = 750000}
    let Chopin = {name ="Frederic Chopin"; listeners = 1200000}
    let Barber = {name = "Samuel Barber"; listeners = 371000}
    let artists = [|Satie; Chopin; Barber|]
    let result = orderArtistsByListenersCount artists
    match result with
    | Success s ->
        Assert.Equal(s.[0], Chopin)
        Assert.Equal(s.[1], Satie)
        Assert.Equal(s.[2], Barber)
    | Failure _ -> Assert.True(false)

Looks like we’re fine now but in fact, nothing stops another developer from implementing a function which just satisfies the magic numbers provided in test.

let orderArtistsByListenersCount artists =
    let Satie = {name = "Erik Satie"; listeners = 750000}
    let Chopin = {name ="Frederic Chopin"; listeners = 1200000}
    let Barber = {name = "Samuel Barber"; listeners = 371000}
    let artists = [|Chopin; Satie; Barber|]
    Success(artists)

Actually, nothing verifies sortedness of an array.

Property-based testing with FsCheck

On the contrary, property-based testing verifies whether our result satisfies the properties that we define. For example, the below code checks whether each item of the sorted array has bigger or same amount of listeners as the next item

open FsCheck.Xunit

let ``pairs from collection should be ordered`` orderFn artists =
    let orderedArtists = artists |> orderFn
    match orderedArtists with
    | Success s -> s |> Array.pairwise |> Array.forall (fun (x,y) -> x.listeners >= y.listeners)
    | Failure _ -> false

[<Property>]
let pairwise x =
    ``pairs from collection should be ordered`` orderArtistsByListenersCount x

What FsCheck does is that it generates several hundred random inputs to test that the property is satisfied upon all possible input values.

Next example checks whether the result array contains same items as the input array.

let ``should be permutation of original elements`` orderFn artists =
    let orderedArtists = artists |> orderFn
    match orderedArtists with
    | Success s -> s |> List.ofArray |> isPermutationOf (List.ofArray artists)
    | Failure _ -> false

[<Property>]
let isPermutation x =
    ``should be permutation of original elements`` orderArtistsByListenersCount x

Implementation of isPermutation can be found here.

Conclusion

I hope those who had the first encounter with F# today didn’t find the uncommon syntax too complex to appreciate such benefits of language as type providers or easy unit testing due to function composition. And I also hope that those who are already on a solid ground found the technique of railway oriented programming quite useful.