Skip to content

The Standard Library🔗

  • Like Python, Go follows a "batteries included" philosophy, offering a rich standard library that addresses modern programming needs.
  • Read : Go Documentation

io and Friends🔗

  • The io package is central to Go's I/O operations, defining interfaces like io.Reader and io.Writer.
  • io.Reader has the Read method, which modifies a provided byte slice and returns the number of bytes read.
  • io.Writer has the Write method, which writes bytes to a destination. It returns number of bytes written and error if something went wrong.
  • These interfaces are widely used for working with files, network connections, and streams.
type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

Efficient Buffer Usage🔗

func countLetters(r io.Reader) (map[string]int, error) {
    buf := make([]byte, 2048)
    out := map[string]int{}
    for {
        n, err := r.Read(buf)
        for _, b := range buf[:n] {
            if (b >= 'A' && b <= 'Z') || (b >= 'a' && b <= 'z') {
                out[string(b)]++
            }
        }
        if err == io.EOF {
            return out, nil
        }
        if err != nil {
            return nil, err
        }
    }
}
  • Using a reusable buffer in Read prevents excessive memory allocations and garbage collection overhead.
// since io.Reader/io.Writer are very simple interfaces that they can be reimplemented in many ways.
s := "The quick brown fox jumped over the lazy dog"
sr := strings.NewReader(s)
counts, err := countLetters(sr)
if err != nil {
    return err
}
fmt.Println(counts)
  • Functions like io.Copy facilitate easy data transfer between io.Reader and io.Writer implementations.
  • Helper functions include:
    • io.MultiReader (reads sequentially from multiple readers)
    • io.MultiWriter (writes to multiple writers simultaneously)
    • io.LimitReader (restricts the number of bytes read from a reader)

Working with Files🔗

  • The os package provides file handling functions like os.Open, os.Create, and os.WriteFile.
  • The bufio package offers buffered I/O for efficient reading and writing.
  • gzip.NewReader wraps an io.Reader to handle gzip-compressed files.

Closing Resources Properly🔗

  • io.Closer interface defines the Close method for resource cleanup.
  • defer f.Close() ensures files are closed after use, preventing resource leaks.
  • Avoid deferring Close in loops to prevent excessive open file handles.

Seeking in Streams🔗

  • io.Seeker provides Seek(offset int64, whence int) (int64, error) for random access in files.
  • The whence parameter should ideally have been a custom type instead of int for clarity.

Combining Interfaces🔗

  • Go defines composite interfaces like io.ReadWriter, io.ReadCloser, and io.ReadWriteSeeker to provide combined functionalities.
  • These interfaces make functions more reusable and compatible with various implementations.

Additional Utils🔗

  • io.ReadAll reads an entire io.Reader into a byte slice.
  • io.NopCloser wraps an io.Reader to satisfy the io.ReadCloser interface, implementing a no-op Close method.
  • The os.ReadFile and os.WriteFile functions handle full-file reads and writes but should be used cautiously with large files.

Go’s io package and related utilities exemplify simplicity and flexibility. By leveraging these standard interfaces and functions, developers can write modular, efficient, and idiomatic Go programs that interact seamlessly with different data sources and sinks.

time🔗

  • time package provides two main types time.Duration and time.Time
  • time.Duration represents a period of time based on int64. It can represent nanoseconds, microseconds, milliseconds, seconds, minutes and hour.
d := 2 * time.Hour + 30 * time.Minute // d is of type time.Duration
  • Go defines a sensible string format, a series of numbers that can be parsed using time.Duration with the time.ParseDuration. Like 300ms, -1.5h or 2h45m. Go Standard Library Documentation
  • time.Time type represents a moment in time complete with timezone.
  • time.Now provides a reference to current time returning a time.Time instance.
  • time.Parse function converts from a string to a time.Time, while the Format method converts a time.Time to a string. Constants Format
t, err := time.Parse("2006-01-02 15:04:05 -0700", "2023-03-13 00:00:00 +0000")
if err != nil {
    return err
}
fmt.Println(t.Format("January 2, 2006 at 3:04:05PM MST"))

// March 13, 2023 at 12:00:00AM UTC
  • time.Time instances can compared using After, Before, and Equal methods.
  • Sub/Add method returns a time.Duration representing elapsed/summing time between two time.Time instances.

Monotonic Time🔗

  • OS tracks two types of time tracking
    • the wall clock which corresponds to the current time.
    • monotonic clock which counts up from time the computer was booted.
  • Reason is tracking two times is because wall clock is not consistent and changes based on Daylight Savings, Leap Seconds, NTP Updates.
  • Go invisibly uses monotonic time to track elapsed time whenever a timer is set or time.Time instance is created.

Timers and Timeout🔗

  • time.After returns a channel that outputs once.
  • time.Tick returns a new value every time specified with time.Duration. NOTE: be careful to use this as underlying time.Ticker can not be shut down. Use the time.NewTicker instead.
  • time.AfterFunc triggers a specific function post some time.Duration

encoding/json🔗

  • REST APIs have made JSON as standard way to communicate between services.
  • Marshalling : converting a Go data type to and encoding
  • Unmarshalling : converting to a Go data type.

Using Struct Tags to Add Metadata🔗

  • Struct tags define how Go struct fields map to JSON fields.
  • Syntax: json:"field_name”
  • If no struct tag is provided, JSON field name default to the Go struct field names.
  • JSON field names are case-insensitive when unmarshalling.
  • To ignore a field : json:"-"
  • To omit empty values: json:"field_name,omitempty"
  • go vet can validate struct tags but the compiler does not.
{
    "id":"12345",
    "date_ordered":"2020-05-01T13:01:02Z",
    "customer_id":"3",
    "items":[{"id":"xyz123","name":"Thing 1"},{"id":"abc789","name":"Thing 2"}]
}
  • To map above data we can create data type as
type Order struct {
    ID            string        `json:"id"`
    DateOrdered   time.Time     `json:"date_ordered"`
    CustomerID    string        `json:"customer_id"`
    Items         []Item        `json:"items"`
}

type Item struct {
    ID   string `json:"id"`
    Name string `json:"name"`
}

Unmarshaling and Marshaling🔗

  • Unmarshalling
var o Order
err := json.Unmarshal([]byte(data), &o)
if err != nil {
    return err
}
  • Marshalling
out, err := json.Marshal(o)
  • Uses reflection to read struct tags dynamically.

JSON, Readers, and Writers🔗

  • json.Marshal and json.Unmarshal work with byte slices
  • Efficient handling via json.Decoder (for reading) and json.Encoder (for writing), which work with io.Reader and io.Writer
  • Example of encoding JSON to a file
err = json.NewEncoder(tmpFile).Encode(toFile)
  • Example decoding JSON from a file
err = json.NewDecoder(tmpFile2).Decode(&fromFile)

Encoding and Decoding JSON Streams🔗

  • Used for handling multiple JSON objects in a stream
  • Decoding JSON streams
var t struct {
    Name string `json:"name"`
    Age  int    `json:"age"`
}
dec := json.NewDecoder(strings.NewReader(streamData))
for {
    err := dec.Decode(&t)
    if err != nil {
        if errors.Is(err, io.EOF) {
            break
        }
        panic(err)
    }
}
  • Encoding multiple JSON objects
var b bytes.Buffer
enc := json.NewEncoder(&b)
for _, input := range allInputs {
    t := process(input)
    err = enc.Encode(t)
    if err != nil {
        panic(err)
    }
}

Custom JSON Parsing🔗

  • Needed when JSON format doesn’t align with Go’s built-in types.
  • Example Custom time format handling
type RFC822ZTime struct {
    time.Time
}

func (rt RFC822ZTime) MarshalJSON() ([]byte, error) {
    out := rt.Time.Format(time.RFC822Z)
    return []byte(`"` + out + `"`), nil
}

func (rt *RFC822ZTime) UnmarshalJSON(b []byte) error {
    if string(b) == "null" {
        return nil
    }
    t, err := time.Parse(`"`+time.RFC822Z+`"`, string(b))
    if err != nil {
        return err
    }
    *rt = RFC822ZTime{t}
    return nil
}
  • Example Struct embedding to override marshalling behaviour
type Order struct {
    ID          string    `json:"id"`
    Items       []Item    `json:"items"`
    DateOrdered time.Time `json:"date_ordered"`
    CustomerID  string    `json:"customer_id"`
}

func (o Order) MarshalJSON() ([]byte, error) {
    type Dup Order
    tmp := struct {
        DateOrdered string `json:"date_ordered"`
        Dup
    }{
        Dup: (Dup)(o),
    }
    tmp.DateOrdered = o.DateOrdered.Format(time.RFC822Z)
    return json.Marshal(tmp)
}

net/http🔗

The Client🔗

  • http.Client handles HTTP requests and responses.
  • Avoid using http.DefaultClient in production (no timeout).
  • Create a custom http.Client
client := &http.Client{
  Timeout: 30 * time.Second
}
  • Use http.NewRequestWithContext to create a request.
  • Set request headers using req.Header.Add().
  • Use client.Do(req) to send the request.
  • Read response data with json.NewDecoder(res.Body).Decode(&data)
// getting a req instance
req, err := http.NewRequestWithContext(context.Background(),
    http.MethodGet, "https://jsonplaceholder.typicode.com/todos/1", nil)
if err != nil {
    panic(err)
}

// adding headers
req.Header.Add("X-My-Client", "Learning Go")
res, err := client.Do(req)
if err != nil {
    panic(err)
}

// decoding response
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
    panic(fmt.Sprintf("unexpected status: got %v", res.Status))
}
fmt.Println(res.Header.Get("Content-Type"))
var data struct {
    UserID    int    `json:"userId"`
    ID        int    `json:"id"`
    Title     string `json:"title"`
    Completed bool   `json:"completed"`
}
err = json.NewDecoder(res.Body).Decode(&data)
if err != nil {
    panic(err)
}
fmt.Printf("%+v\n", data)

The Server🔗

  • http.Server handles HTTP requests.
  • Implements http.Handler interface:
type Handler interface {
    ServeHTTP(http.ResponseWriter, *http.Request)
}

s := http.Server{
    Addr:         ":8080",  // default 80
    ReadTimeout:  30 * time.Second,
    WriteTimeout: 90 * time.Second,
    Handler:      HelloHandler{},   // mux can be used
}
err := s.ListenAndServe()
if err != nil {
    if err != http.ErrServerClosed {
        panic(err)
    }
}
  • http.ResponseWriter methods includes (in order):
    • Header() http.Header
    • Write([]byte) (int, error)
    • WriteHeader(statusCode int)

Request Routing🔗

  • *http.ServeMux is used for routing requests
  • http.NewServeMux fuction creates a new instance of ServeMux which meets the http.Handler interface, so can be assigned to Handler field in http.Server.
mux.HandleFunc("/hello", func(w http.ResponseWrite, r *http.Request)) {
  w.Write([] byte("Hello!\n"));
}
  • Go 1.22 extends the path syntax to optionally allow HTTP verbs and path wildcard variables.
mux.HandleFunc("GET /hello/{name}", func(w http.ResponseWriter, r *http.Request) {
    name := r.PathValue("name")
    w.Write([]byte(fmt.Sprintf("Hello, %s!\n", name)))
})
  • *http.ServeMux dispatches request to http.Handler instances and implements them. We can create *http.ServeMux instance with multiple related request and register it with a parent *http.ServeMux
person := http.NewServeMux()
person.HandleFunc("/greet", func(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("greetings!\n"))
})
dog := http.NewServeMux()
dog.HandleFunc("/greet", func(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("good puppy!\n"))
})
mux := http.NewServeMux()
mux.Handle("/person/", http.StripPrefix("/person", person))
mux.Handle("/dog/", http.StripPrefix("/dog", dog))
  • /person/greet is handled by handler attached to person, while /dog/greet is handled by handlers attached to dog

Middleware🔗

  • Middleware takes http.Handler instance and returns http.Handler (usually a closure)
  • Its usually used to plugin some transformation or validation in requests.
// example-request timer middleware
func RequestTimer(h http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        h.ServeHTTP(w, r)
        dur := time.Since(start)
        slog.Info("request time",
            "path", r.URL.Path,
            "duration", dur)
    })
}

// using it
mux.Handle("/hello", RequestTimer(
    http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello!\n"))
    })))
  • Since *http.ServeMux implements the http.Handler interface, a set of middlewares can be applied to all registered handlers.
terribleSecurity := TerribleSecurityProvider("GOPHER")
wrappedMux := terribleSecurity(RequestTimer(mux))
s := http.Server{
    Addr:    ":8080",
    Handler: wrappedMux,
}
  • Third Party modules to enhance server
    • alice allows function chains for middlewares
    • gorilla mux and chi both are very good request routers
    • Echo and Gin web frameworks implements their own handler and middleware patterns

Response Controller🔗

  • Backward Compatibility Challenge:

    • Modifying interfaces breaks backward compatibility in Go
    • New methods cannot be added to existing interfaces without breaking existing implementations

  • Tradition Approach - Using Additional Interfaces

    • Example: http.ResponseWriter could not be modified directly
    • Instead, optional functionality added through new interfaces like http.Flusher and http.Hijacker
    • This approach makes hard to discover new interfaces and requires verbose type assertions

  • New Approach - Concrete Wrapper Type

    • In Go 1.20, to extend http.ResponseWriter without breaking compatibility
    • http.NewResponseController(rw) returns a http.ResponseController that wraps http.ResponseWriter
    • New optional methods are exposed via this wrapper, avoiding interface modification.

  • Checking for Optional Methods using Error Handling

    • Instead of type assertions, optional functionality is checked using error comparison:
    • go err = rc.Flush() if err != nil && !errors.Is(err, http.ErrNotSupported) { /* handle error */ }
    • If Flush is unsupported, program handles it gracefully.
  • Advantages of http.ResponseController
    • New methods can be added without breaking existing implementations.
    • Makes new functionality discoverable and easy to use.
    • Provides a standardized way to check for optional methods using error handling
  • Future Use
    • More optional methods (like SetReadDeadline, SetWriteDeadline) are added via http.ResponseController.
    • This pattern will likely be used for future extensions of http.ResponseWriter.

Structured Logging🔗

  • The Go standard library initially included the log package for simple logging, which lacked support for structured logs.
  • Structured logs use a consistent format for each entry, facilitating automated processing and analysis.

Challenges with Unstructured Logging🔗

  • Modern web services handle millions of users simultaneously, necessitating automated log analysis.
  • Unstructured logs complicate pattern recognition and anomaly detection due to inconsistent formatting.

Introduction of log/slog Package🔗

  • Go 1.21 introduced the log/slog package to address structured logging needs.
  • This addition promotes consistency and interoperability among Go modules.

Advantages of Standardized Structured Logging🔗

  • Prior to log/slog, various third-party loggers like zap, logrus, and go-kit log offered structured logging, leading to fragmentation.
  • A unified standard library logger simplifies integration and control over log outputs and levels.
  • Structured logging was introduced as a separate package (log/slog) rather than modifying the existing log package to maintain clear and distinct APIs.
  • The API is scalable, starting with simple default loggers and allowing for advanced configurations.

Basic Usage of log/slog🔗

  • Provides functions like slog.Debug, slog.Info, slog.Warn, and slog.Error for logging at various levels.
  • Default logger output includes timestamp, log level, and message.
  • Supports adding custom key-value pairs to log entries for enhanced context.
userID := "fred"
loginCount := 20
slog.Info("user login", "id", userID, "login_count", loginCount)

// output
2023/04/20 23:36:38 INFO user login id=fred login_count=20

Customizing Log Output Formats:🔗

  • Allows switching from text to JSON format for logs.
options := &slog.HandlerOptions{Level: slog.LevelDebug}
handler := slog.NewJSONHandler(os.Stderr, options)
mySlog := slog.New(handler)
lastLogin := time.Date(2023, 01, 01, 11, 50, 00, 00, time.UTC)
mySlog.Debug("debug message", "id", userID, "last_login", lastLogin)

// ouput
{"time":"2023-04-22T23:30:01.170243-04:00","level":"DEBUG","msg":"debug message","id":"fred","last_login":"2023-01-01T11:50:00Z"}

Performance Consideration🔗

  • To optimize performance and reduce allocations, use LogAttrs with predefined attribute types.
mySlog.LogAttrs(ctx, slog.LevelInfo, "faster logging",
            slog.String("id", userID),
            slog.Time("last_login", lastLogin))

Interoperability with Existing log Package:🔗

  • The log package remains supported; log/slog offers enhanced features without deprecating existing functionality.
  • Bridging between log and slog is possible using slog.NewLogLogger.
myLog := slog.NewLogLogger(mySlog.Handler(), slog.LevelDebug)
myLog.Println("using the mySlog Handler")

// 
{"time":"2023-04-22T23:30:01.170269-04:00","level":"DEBUG","msg":"using the mySlog Handler"}

Additional Features of log/slog:

  • Supports dynamic logging levels and context integration.
  • Offers value grouping and common header creation for logs.
  • Detailed API documentation provides further insights into advanced functionalities.