Ｇo - Parsing JSON Data Streams Into Structs

Problem: You want to parse JSON data from a stream.

Solution: Create structs to contain the JSON data. Create a decoder using NewDecoder in the encoding/json package, then call Decode on the decoder to decode data into the structs.

 

In this first JSON file you have an array of three JSON objects (part of the data is truncated to make it easier to read):

[
    {
        "name": "Luke  Skywalker",
        "height": "172",
        "mass": "77",
        "hair_color": "blond",
        "skin_color": "fair",
        "eye_color": "blue",
        "birth_year": "19BBY",
        "gender": "male"
    },
    {
        "name": "C - 3PO",
        "height": "167",
        "mass": "75",
        "hair_color": "n/a",
        "skin_color": "gold",
        "eye_color": "yellow",
        "birth_year": "112BBY",
        "gender": "n/a"
    },
    {
        "name": "R2 - D2",
        "height": "96",
        "mass": "32",
        "hair_color": "n/a",
        "skin_color": "white,  blue",
        "eye_color": "red",
        "birth_year": "33BBY",
        "gender": "n/a"
    }
]

To read this, you can use Unmarshal by unmarshalling into an array of Person structs:

type Person struct {
    Name string `json:"name"`
    Height string `json:"height"`
    Mass string `json:"mass"`
    HairColor string `json:"hair_color"`
    SkinColor string `json:"skin_color"`
    EyeColor string `json:"eye_color"`
    BirthYear string `json:"birth_year"`
    Gender string `json:"gender"`
    Homeworld string `json:"homeworld"`
    Films []string `json:"films"`
    Species []string `json:"species"`
    Vehicles []string `json:"vehicles"`
    Starships []string `json:"starships"`
    Created time.Time `json:"created"`
    Edited time.Time `json:"edited"`
    URL string `json:"url"`
}

func chkPrtErr(err error, msg string) {
    if err != nil {
        log.Printf("%s: %s\n", msg, err)
    }
}

func unmarshalStructArray() (people []Person) {
    file, err := os.Open("people.json")
    chkPrtErr(err, "Error opening json file")
    defer file.Close()

    data, err := io.ReadAll(file)
    chkPrtErr(err, "Error reading json data")

    err = json.Unmarshal(data, &people)
    chkPrtErr(err, "Error unmarshalling json data")

    return
}

This will result in an output like this:

[]json.Person{
    {
        Name:       "Luke  Skywalker",
        Height:     "172",
        Mass:       "77",
        HairColor:  "blond",
        SkinColor:  "fair",
        EyeColor:   "blue",
        BirthYear:  "19BBY",
        Gender:     "male",
        Homeworld:  "",
        Films:      nil,
        Species:    nil,
        Vehicles:   nil,
        Starships:  nil,
        Created:    time.Date(1,  time.January,  1,  0,  0,  0,  0,  time.UTC),
        Edited:     time.Date(1,  time.January,  1,  0,  0,  0,  0,  time.UTC),
        URL:        "",
    },
    {
        Name:       "C - 3PO",
        Height:     "167",
        Mass:       "75",
        HairColor:  "n/a",
        SkinColor:  "gold",
        EyeColor:   "yellow",
        BirthYear:  "112BBY",
        Gender:     "n/a",
        Homeworld:  "",
        Films:      nil,
        Species:    nil,
        Vehicles:   nil,
        Starships:  nil,
        Created:    time.Date(1,  time.January,  1,  0,  0,  0,  0,  time.UTC),
        Edited:     time.Date(1,  time.January,  1,  0,  0,  0,  0,  time.UTC),
        URL:        "",
    },
    {
        Name:       "R2 - D2",
        Height:     "96",
        Mass:       "32",
        HairColor:  "n/a",
        SkinColor:  "white,  blue",
        EyeColor:   "red",
        BirthYear:  "33BBY",
        Gender:     "n/a",
        Homeworld:  "",
        Films:      nil,
        Species:    nil,
        Vehicles:   nil,
        Starships:  nil,
        Created:    time.Date(1,  time.January,  1,  0,  0,  0,  0,  time.UTC),
        Edited:     time.Date(1,  time.January,  1,  0,  0,  0,  0,  time.UTC),
        URL:        "",
    },
}

This is an array of Person structs, which you get after unmarshalling a single JSON array.

 

However, when you get a stream of JSON objects, this is no longer possible. Here is another JSON file, one that is representative of a JSON data stream:

{
    "name": "Luke  Skywalker",
    "height": "172",
    "mass": "77",
    "hair_color": "blond",
    "skin_color": "fair",
    "eye_color": "blue",
    "birth_year": "19BBY",
    "gender": "male"
} 
{
    "name": "C - 3PO",
    "height": "167",
    "mass": "75",
    "hair_color": "n/a",
    "skin_color": "gold",
    "eye_color": "yellow",
    "birth_year": "112BBY",
    "gender": "n/a"
} 
{
    "name": "R2 - D2",
    "height": "96",
    "mass": "32",
    "hair_color": "n/a",
    "skin_color": "white,  blue",
    "eye_color": "red",
    "birth_year": "33BBY",
    "gender": "n/a"
}

Notice that this is not a single JSON object but three consecutive JSON objects. This is no longer a valid JSON file, but it’s something you can get when you read the Body of a http.Response struct. If you try to read this using Unmarshal you will get an error:

Error  unmarshalling  json  data:  invalid  character  '{'  after  top - level  value

However, you can parse it by decoding it using a Decoder:

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "os"
    "time"

    "github.com/kr/pretty"
)

type Person struct {
    Name      string    `json:"name"`
    Height    string    `json:"height"`
    Mass      string    `json:"mass"`
    HairColor string    `json:"hair_color"`
    SkinColor string    `json:"skin_color"`
    EyeColor  string    `json:"eye_color"`
    BirthYear string    `json:"birth_year"`
    Gender    string    `json:"gender"`
    Homeworld string    `json:"homeworld"`
    Films     []string  `json:"films"`
    Species   []string  `json:"species"`
    Vehicles  []string  `json:"vehicles"`
    Starships []string  `json:"starships"`
    Created   time.Time `json:"created"`
    Edited    time.Time `json:"edited"`
    URL       string    `json:"url"`
}

func decode(p chan Person) {
    file, err := os.Open("people_stream.json")
    if err != nil {
        log.Println("Error  opening  json  file:", err)
    }
    defer file.Close()

    decoder := json.NewDecoder(file)
    for {
        var person Person
        err = decoder.Decode(&person)
        if err == io.EOF {
            break
        }
        if err != nil {
            log.Println("Error  decoding  json  data:", err)
            break
        }
        p <- person
    }
    close(p)
}

First, you create a decoder using json.NewDecoder and passing it the reader, in this case, it’s the file you read from. Then while you’re looping in the for loop, you call Decode on the decoder, passing it the struct you want to store the data in. If all goes well, every time it loops, a new Person struct instance is created from the data. You can use the data then. If there is no more data in the reader, i.e., you hit io.EOF , you’ll break from the for loop.

In the case of the preceding code, you pass in a channel, in which you store the Person struct instance in every loop. When you’re done reading all the JSON in the file, you’ll close the channel:

func main() {
    p := make(chan Person)
    go decode(p)
    for {
        person, ok := <-p
        if ok {
            fmt.Printf("%# v\n", pretty.Formatter(person))
        } else {
            break
        }
    }
}

Here’s the output from the code:

zzh@ZZHPC:/zdata/MyPrograms/Go/study$ go run main.go
main.Person{
    Name:      "Luke  Skywalker",
    Height:    "172",
    Mass:      "77",
    HairColor: "blond",
    SkinColor: "fair",
    EyeColor:  "blue",
    BirthYear: "19BBY",
    Gender:    "male",
    Homeworld: "",
    Films:     nil,
    Species:   nil,
    Vehicles:  nil,
    Starships: nil,
    Created:   time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC),
    Edited:    time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC),
    URL:       "",
}
main.Person{
    Name:      "C - 3PO",
    Height:    "167",
    Mass:      "75",
    HairColor: "n/a",
    SkinColor: "gold",
    EyeColor:  "yellow",
    BirthYear: "112BBY",
    Gender:    "n/a",
    Homeworld: "",
    Films:     nil,
    Species:   nil,
    Vehicles:  nil,
    Starships: nil,
    Created:   time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC),
    Edited:    time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC),
    URL:       "",
}
main.Person{
    Name:      "R2 - D2",
    Height:    "96",
    Mass:      "32",
    HairColor: "n/a",
    SkinColor: "white,  blue",
    EyeColor:  "red",
    BirthYear: "33BBY",
    Gender:    "n/a",
    Homeworld: "",
    Films:     nil,
    Species:   nil,
    Vehicles:  nil,
    Starships: nil,
    Created:   time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC),
    Edited:    time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC),
    URL:       "",
}

You can see that three Person structs are being printed here, one after another, as opposed to the earlier one that was an array of Person structs.

 

A question that sometimes arises is when should you use Unmarshal and when should you use Decode ?
Unmarshal is easier to use for a single JSON object, but it won’t work when you have a stream of them coming in from a reader. Also, its simplicity means it’s not as flexible; you just get the whole JSON data at a go.
Decode , on the other hand, works well for both single JSON objects and streaming JSON data. Also, with Decode you can do stuff with the JSON at a finer level without needing to get the entire JSON data out first. This is because you can inspect the JSON as it comes in, even at a token level. The only slight drawback is that it is more verbose.
In addition, Decode is a bit faster.