Quantcast
Channel: The Go Blog
Viewing all 267 articles
Browse latest View live

The Gopher Gala is the first worldwide Go hackathon

$
0
0

The Gopher Gala is the first Go hackathon at a global scale and will take place from January 23rd through the 25th. The event is organized by the community, supported by the Go team, and sponsored by Google among others.

Fancy Gopher, byRenée French

You can read about the rules of the hackathonhere, but if you know aboutRails Rumble orNode Knockout you already have a pretty good idea of what to expect.

During this event gophers from all around the globe will form teams to build great applications using Go as the main tool. Afterwards, all the participants will vote for their favorite applications and the 20 highest voted applications will be ranked by a jury of renowned names from the community, including some members of the Go core team. More information on the judging phase can be foundhere.

And in case you needed one more reason toget involved there will be prizes!

"What prizes?" you ask. Well, that’s asecret until January 10th but we’re pretty sure you won’t be disappointed.


GothamGo: gophers in the big apple

$
0
0

Last November more than two hundred gophers from all across the United States got together for the first full-day Go conference in New York City.

The diverse speaker lineup included university students, industry experts, and Go team members.

And good news, everybody! All the talks were recorded and are available:

Two more talks come from the Go meetup in New York City, which met the day before GothamGo:

  • Benchmarking GobyBrian Bulkowski - the founder of Aerospike talks about profiling tools for Go and Linux, and micro benchmarks for goroutines, channels, buffers, and and other Go features.
  • Go Static Analysis Toolsby Alan Donovan - a member of the Go team at Google NY gives a guided tour of several static analysis tools designed to help Go programmers understand, navigate , and refactor their code.

Make sure to have a look at all of those in preparation for the FOSDEM Go devroom FOSDEM in Brussels (Belgium) and gophercon.in in Bengaluru (India).

Errors are values

$
0
0

A common point of discussion among Go programmers, especially those new to the language, is how to handle errors. The conversation often turns into a lament at the number of times the sequence

if err != nil {    return err}

shows up. We recently scanned all the open source projects we could find and discovered that this snippet occurs only once per page or two, less often than some would have you believe. Still, if the perception persists that one must type

if err != nil

all the time, something must be wrong, and the obvious target is Go itself.

This is unfortunate, misleading, and easily corrected. Perhaps what is happening is that programmers new to Go ask,"How does one handle errors?", learn this pattern, and stop there. In other languages, one might use a try-catch block or other such mechanism to handle errors. Therefore, the programmer thinks, when I would have used a try-catch in my old language, I will just type iferr!=nil in Go. Over time the Go code collects many such snippets, and the result feels clumsy.

Regardless of whether this explanation fits, it is clear that these Go programmers miss a fundamental point about errors:Errors are values.

Values can be programmed, and since errors are values, errors can be programmed.

Of course a common statement involving an error value is to test whether it is nil, but there are countless other things one can do with an error value, and application of some of those other things can make your program better, eliminating much of the boilerplate that arises if every error is checked with a rote if statement.

Here's a simple example from the bufio package'sScanner type. Its Scan method performs the underlying I/O, which can of course lead to an error. Yet the Scan method does not expose an error at all. Instead, it returns a boolean, and a separate method, to be run at the end of the scan, reports whether an error occurred. Client code looks like this:

scanner := bufio.NewScanner(input)for scanner.Scan() {    token := scanner.Text()    // process token}if err := scanner.Err(); err != nil {    // process the error}

Sure, there is a nil check for an error, but it appears and executes only once. The Scan method could instead have been defined as

func (s *Scanner) Scan() (token []byte, error)

and then the example user code might be (depending on how the token is retrieved),

scanner := bufio.NewScanner(input)for {    token, err := scanner.Scan()    if err != nil {        return err // or maybe break    }    // process token}

This isn't very different, but there is one important distinction. In this code, the client must check for an error on every iteration, but in the real Scanner API, the error handling is abstracted away from the key API element, which is iterating over tokens. With the real API, the client's code therefore feels more natural: loop until done, then worry about errors. Error handling does not obscure the flow of control.

Under the covers what's happening, of course, is that as soon as Scan encounters an I/O error, it records it and returns false. A separate method, Err, reports the error value when the client asks. Trivial though this is, it's not the same as putting

if err != nil

everywhere or asking the client to check for an error after every token. It's programming with error values. Simple programming, yes, but programming nonetheless.

It's worth stressing that whatever the design, it's critical that the program check the errors however they are exposed. The discussion here is not about how to avoid checking errors, it's about using the language to handle errors with grace.

The topic of repetitive error-checking code arose when I attended the autumn 2014 GoCon in Tokyo. An enthusiastic gopher, who goes by @jxck_ on Twitter, echoed the familiar lament about error checking. He had some code that looked schematically like this:

_, err = fd.Write(p0[a:b])if err != nil {    return err}_, err = fd.Write(p1[c:d])if err != nil {    return err}_, err = fd.Write(p2[e:f])if err != nil {    return err}// and so on

It is very repetitive. In the real code, which was longer, there is more going on so it's not easy to just refactor this using a helper function, but in this idealized form, a function literal closing over the error variable would help:

var err errorwrite := func(buf []byte) {    if err != nil {        return    }    _, err = w.Write(buf)}write(p0[a:b])write(p1[c:d])write(p2[e:f])// and so onif err != nil {    return err}

This pattern works well, but requires a closure in each function doing the writes; a separate helper function is clumsier to use because the err variable needs to be maintained across calls (try it).

We can make this cleaner, more general, and reusable by borrowing the idea from theScan method above. I mentioned this technique in our discussion but @jxck_ didn't see how to apply it. After a long exchange, hampered somewhat by a language barrier, I asked if I could just borrow his laptop and show him by typing some code.

I defined an object called an errWriter, something like this:

type errWriter struct {    w   io.Writer    err error}

and gave it one method, write. It doesn't need to have the standard Write signature, and it's lower-cased in part to highlight the distinction. The write method calls the Write method of the underlying Writer and records the first error for future reference:

func (ew *errWriter) write(buf []byte) {    if ew.err != nil {        return    }    _, ew.err = ew.w.Write(buf)}

As soon as an error occurs, the write method becomes a no-op but the error value is saved.

Given the errWriter type and its write method, the code above can be refactored:

ew := &errWriter{w: fd}ew.write(p0[a:b])ew.write(p1[c:d])ew.write(p2[e:f])// and so onif ew.err != nil {    return ew.err}

This is cleaner, even compared to the use of a closure, and also makes the actual sequence of writes being done easier to see on the page. There is no clutter any more. Programming with error values (and interfaces) has made the code nicer.

It's likely that some other piece of code in the same package can build on this idea, or even use errWriter directly.

Also, once errWriter exists, there's more it could do to help, especially in less artificial examples. It could accumulate the byte count. It could coalesce writes into a single buffer that can then be transmitted atomically. And much more.

In fact, this pattern appears often in the standard library. The archive/zip andnet/http packages use it. More salient to this discussion, the bufio package's Writer is actually an implementation of the errWriter idea. Although bufio.Writer.Write returns an error, that is mostly about honoring the io.Writer interface. The Write method of bufio.Writer behaves just like our errWriter.write method above, with Flush reporting the error, so our example could be written like this:

b := &bufio.NewWriter(fd)b.Write(p0[a:b])b.Write(p1[c:d])b.Write(p2[e:f])// and so onif b.Flush() != nil {    return b.Flush()}

There is one significant drawback to this approach, at least for some applications: there is no way to know how much of the processing completed before the error occurred. If that information is important, a more fine-grained approach is necessary. Often, though, an all-or-nothing check at the end is sufficient.

We've looked at just one technique for avoiding repetitive error handling code. Keep in mind that the use of errWriter or bufio.Writer isn't the only way to simplify error handling, and this approach is not suitable for all situations. The key lesson, however, is that errors are values and the full power of the Go programming language is available for processing them.

Use the language to simplify your error handling.

But remember: Whatever you do, always check your errors!

Finally, for the full story of my interaction with @jxck_, including a little video he recorded, visit his blog.

Package names

$
0
0

Introduction

Go code is organized into packages. Within a package, code can refer to any identifier (name) defined within, while clients of the package may only reference the package's exported types, functions, constants, and variables. Such references always include the package name as a prefix: foo.Bar refers to the exported name Bar in the imported package named foo.

Good package names make code better. A package's name provides context for its contents, making it easier for clients to understand what the package is for and how to use it. The name also helps package maintainers determine what does and does not belong in the package as it evolves. Well-named packages make it easier to find the code you need.

Effective Go providesguidelines for naming packages, types, functions, and variables. This article expands on that discussion and surveys names found in the standard library. It also discusses bad package names and how to fix them.

Package names

Good package names are short and clear. They are lower case, with no under_scores or mixedCaps. They are often simple nouns, such as:

  • time (provides functionality for measuring and displaying time)
  • list (implements a doubly linked list)
  • http (provides HTTP client and server implementations)

The style of names typical of another language might not be idiomatic in a Go program. Here are two examples of names that might be good style in other languages but do not fit well in Go:

  • computeServiceClient
  • priority_queue

A Go package may export several types and functions. For example, a compute package could export a Client type with methods for using the service as well as functions for partitioning a compute task across several clients.

Abbreviate judiciously. Package names may be abbreviated when the abbreviation is familiar to the programmer. Widely-used packages often have compressed names:

  • strconv (string conversion)
  • syscall (system call)
  • fmt (formatted I/O)

On the other hand, if abbreviating a package name makes it ambiguous or unclear, don't do it.

Don't steal good names from the user. Avoid giving a package a name that is commonly used in client code. For example, the buffered I/O package is called bufio, not buf, since buf is a good variable name for a buffer.

Naming package contents

A package name and its contents' names are coupled, since client code uses them together. When designing a package, take the client's point of view.

Avoid stutter. Since client code uses the package name as a prefix when referring to the package contents, the names for those contents need not repeat the package name. The HTTP server provided by the http package is called Server, notHTTPServer. Client code refers to this type as http.Server, so there is no ambiguity.

Simplify function names. When a function in package pkg returns a value of type pkg.Pkg (or*pkg.Pkg), the function name can often omit the type name without confusion:

start := time.Now()                           // start is a time.Timet, err := time.Parse(time.Kitchen, "6:06PM")  // t is a time.Time

A function named New in package pkg returns a value of type pkg.Pkg. This is a standard entry point for client code using that type:

q := list.New()  // q is a *list.List

When a function returns a value of type pkg.T, where T is not Pkg, the function name may include T to make client code easier to understand. A common situation is a package with multiple New-like functions:

d, err := time.ParseDuration("10s")  // d is a time.Durationelapsed := time.Since(start)         // elapsed is a time.Durationticker := time.NewTicker(d)          // ticker is a *time.Tickertimer := time.NewTimer(d)            // timer is a *time.Timer

Types in different packages can have the same name, because from the client's point of view such names are discriminated by the package name. For example, the standard library includes several types named Reader, including jpeg.Reader, bufio.Reader, and csv.Reader. Each package name fits with Reader to yield a good type name.

If you cannot come up with a package name that's a meaningful prefix for the package's contents, the package abstraction boundary may be wrong. Write code that uses your package as a client would, and restructure your packages if the result seems poor. This approach will yield packages that are easier for clients to understand and for the package developers to maintain.

Package paths

A Go package has both a name and a path. The package name is specified in the package statement of its source files; client code uses it as the prefix for the package's exported names. Client code uses the package path when importing the package. By convention, the last element of the package path is the package name:

import ("fmt"                       // package fmt"os/exec"                   // package exec"golang.org/x/net/context"  // package context)

Build tools map package paths onto directories. The go tool uses the GOPATH environment variable to find the source files for path "github.com/user/hello" in directory $GOPATH/src/github.com/user/hello. (This situation should be familiar, of course, but it's important to be clear about the terminology and structure of packages.)

Directories. The standard library uses like directories crypto, container, encoding, and image to group packages for related protocols and algorithms. There is no actual relationship among the packages in one of these directories; a directory just provides a way to arrange the files. Any package can import any other package provided the import does not create a cycle.

Just as types in different packages can have the same name without ambiguity, packages in different directories can have the same name. For example,runtime/pprof provides profiling data in the format expected by the pprof profiling tool, while net/http/pprof provides HTTP endpoints to present profiling data in this format. Client code uses the package path to import the package, so there is no confusion. If a source file needs to import both pprof packages, it canrename one or both locally. When renaming an imported package, the local name should follow the same guidelines as package names (lower case, no under_scores or mixedCaps).

Bad package names

Bad package names make code harder to navigate and maintain. Here are some guidelines for recognizing and fixing bad names.

Avoid meaningless package names. Packages named util, common, or misc provide clients with no sense of what the package contains. This makes it harder for clients to use the package and makes it harder for maintainers to keep the package focused. Over time, they accumulate dependencies that can make compilation significantly and unnecessarily slower, especially in large programs. And since such package names are generic, they are more likely to collide with other packages imported by client code, forcing clients to invent names to distinguish them.

Break up generic packages. To fix such packages, look for types and functions with common name elements and pull them into their own package. For example, if you have

package utilfunc NewStringSet(...string) map[string]bool {...}func SortStringSet(map[string]bool) []string {...}

then client code looks like

set := util.NewStringSet("c", "a", "b")fmt.Println(util.SortStringSet(set))

Pull these functions out of util into a new package, choosing a name that fits the contents:

package stringsetfunc New(...string) map[string]bool {...}func Sort(map[string]bool) []string {...}

then the client code becomes

set := stringset.New("c", "a", "b")fmt.Println(stringset.Sort(set))

Once you've made this change, its easier to see how to improve the new package:

package stringsettype Set map[string]boolfunc New(...string) Set {...}func (s Set) Sort() []string {...}

which yields even simpler client code:

set := stringset.New("c", "a", "b")fmt.Println(set.Sort())

The name of the package is a critical piece of its design. Work to eliminate meaningless package names from your projects.

Don't use a single package for all your APIs. Many well-intentioned programmers put all the interfaces exposed by their program into a single package named api, types, or interfaces, thinking it makes it easier to find the entry points to their code base. This is a mistake. Such packages suffer from the same problems as those named util or common, growing without bound, providing no guidance to users, accumulating dependencies, and colliding with other imports. Break them up, perhaps using directories to separate public packages from implementation.

Avoid unnecessary package name collisions. While packages in different directories may have the same name, packages that are frequently used together should have distinct names. This reduces confusion and the need for local renaming in client code. For the same reason, avoid using the same name as popular standard packages likeio or http.

Conclusion

Package names are central to good naming in Go programs. Take the time to choose good package names and organize your code well. This helps clients understand and use your packages and helps maintainers to grow them gracefully.

Further reading

Testable Examples in Go

$
0
0

Introduction

Godoc examples are snippets of Go code that are displayed as package documentation and that are verified by running them as tests. They can also be run by a user visiting the godoc web page for the package and clicking the associated "Run" button.

Having executable documentation for a package guarantees that the information will not go out of date as the API changes.

The standard library includes many such examples (see the strings package, for instance).

This article explains how to write your own example functions.

Examples are tests

Examples are compiled (and optionally executed) as part of a package's test suite.

As with typical tests, examples are functions that reside in a package's_test.go files. Unlike normal test functions, though, example functions take no arguments and begin with the word Example instead of Test.

The stringutil package is part of the Go example repository. Here's an example that demonstrates its Reverse function:

package stringutil_testimport ("fmt""github.com/golang/example/stringutil")func ExampleReverse() {    fmt.Println(stringutil.Reverse("hello"))    // Output: olleh}

This code might live in example_test.go in the stringutil directory.

Godoc will present this example alongside the Reverse function's documentation:

Running the package's test suite, we can see the example function is executed with no further arrangement from us:

$ go test -v=== RUN TestReverse--- PASS: TestReverse (0.00s)=== RUN: ExampleReverse--- PASS: ExampleReverse (0.00s)PASSok      github.com/golang/example/stringutil    0.009s

Output comments

What does it mean that the ExampleReverse function "passes"?

As it executes the example, the testing framework captures data written to standard output and then compares the output against the example's "Output:" comment. The test passes if the test's output matches its output comment.

To see a failing example we can change the output comment text to something obviously incorrect

func ExampleReverse() {    fmt.Println(stringutil.Reverse("hello"))    // Output: golly}

and run the tests again:

$ go test--- FAIL: ExampleReverse (0.00s)got:ollehwant:gollyFAIL

If we remove the output comment entirely

func ExampleReverse() {    fmt.Println(stringutil.Reverse("hello"))}

then the example function is compiled but not executed:

$ go test -v=== RUN TestReverse--- PASS: TestReverse (0.00s)PASSok      github.com/golang/example/stringutil    0.009s

Examples without output comments are useful for demonstrating code that cannot run as unit tests, such as that which accesses the network, while guaranteeing the example at least compiles.

Example function names

Godoc uses a naming convention to associate an example function with a package-level identifier.

func ExampleFoo()     // documents the Foo function or typefunc ExampleBar_Qux() // documents the Qux method of type Barfunc Example()        // documents the package as a whole

Following this convention, godoc displays the ExampleReverse example alongside the documentation for the Reverse function.

Multiple examples can be provided for a given identifier by using a suffix beginning with an underscore followed by a lowercase letter. Each of these examples documents the Reverse function:

func ExampleReverse()func ExampleReverse_second()func ExampleReverse_third()

Larger examples

Sometimes we need more than just a function to write a good example.

For instance, to demonstrate the sort package we should show an implementation of sort.Interface. Since methods cannot be declared inside a function body, the example must include some context in addition to the example function.

To achieve this we can use a "whole file example." A whole file example is a file that ends in _test.go and contains exactly one example function, no test or benchmark functions, and at least one other package-level declaration. When displaying such examples godoc will show the entire file.

Here is a whole file example from the sort package:

package sort_testimport ("fmt""sort")type Person struct {    Name string    Age  int}func (p Person) String() string {    return fmt.Sprintf("%s: %d", p.Name, p.Age)}// ByAge implements sort.Interface for []Person based on// the Age field.type ByAge []Personfunc (a ByAge) Len() int           { return len(a) }func (a ByAge) Swap(i, j int)      { a[i], a[j] = a[j], a[i] }func (a ByAge) Less(i, j int) bool { return a[i].Age < a[j].Age }func Example() {    people := []Person{        {"Bob", 31},        {"John", 42},        {"Michael", 17},        {"Jenny", 26},    }    fmt.Println(people)    sort.Sort(ByAge(people))    fmt.Println(people)    // Output:    // [Bob: 31 John: 42 Michael: 17 Jenny: 26]    // [Michael: 17 Jenny: 26 Bob: 31 John: 42]}

A package can contain multiple whole file examples; one example per file. Take a look at the sort package's source code to see this in practice.

Conclusion

Godoc examples are a great way to write and maintain code as documentation. They also present editable, working, runnable examples your users can build on. Use them!

GopherChina Trip Report

$
0
0

We have known for some time that Go is more popular in China than in any other country. According to Google Trends, most searches for the term “golang” come from The People’s Republic than anywhere else.Others have speculated on the same observation, yet so far we have hadsparse concrete information about the phenomenon.

The first Go conference in China, GopherChina, seemed like an excellent opportunity to explore the situation by putting some Western Gopher feet on Chinese ground. An actual invitation made it real and I decided to accept and give a presentation about gofmt’s impact on software development.

Hello, Shanghai!

The conference took place over an April weekend in Shanghai, in thePuruan Building of the Shanghai Pudong Software Park, easily reachable by subway within an hour or less from Shanghai’s more central parts. Modelled after GopherCon, the conference was single-track, with all talks presented in a conference room that fit about 400 attendees. It was organized by volunteers, lead by Asta Xie, and with robust sponsorship from major industry names. According to the organizers, many more people were hoping to attend than could be accommodated due to space constraints.

The welcoming committee with Asta Xie (2nd from left), the primary organizer.

Each attendee received a bag filled with the obligatory GopherChina t-shirt, various sponsor-related informational brochures, stickers, and the occasional stuffed “something” (no fluffy Gophers, though). At least one 3rd party vendor was advertising technical books, including several original (not translated from English) Go books.

Go books!

On first impression, the average attendee seemed pretty young, which made for an enthusiastic crowd, and the event appeared well run.

With the exception of my talk, all presentations were given in Mandarin and thus were incomprehensible to me. Asta Xie, the primary organizer, assisted with a few simultaneous translations whispered into my ear, and the occasional English slide provided additional clues: “69GB” stands out even without any Mandarin knowledge (more on that below). Consequently, I ended up listening to a handful of presentations only, and instead spent much of my time talking with attendees outside the main conference room. Yet judging from the slides, the quality of most presentations seemed high, comparable with our experience at GopherCon in Denver last year. Each talk got a one hour time slot which allowed for plenty of technical detail, and many (dozens) of questions from an enthusiastic audience.

As expected, many of the presentations were about web services, backends for mobile applications, and so on. Some of the systems appear to be huge by any measure. For instance, a talk by Yang Zhou described a large-scale internal messaging system, used byQihoo 360, a major Chinese software firm, all written in Go. The presentation discussed how his team managed to reduce an original heap size of 69GB (!) and the resulting long GC pauses of 3-6s to more manageable numbers, and how they run millions of goroutines per machine, on a fleet of thousands of machines. A future guest blog post is planned describing this system in more detail.

Packed conference room on Saturday.

In another presentation, Feng Guo fromDaoCloud talked about how they use Go in their company for what they call the “continuous delivery” of applications. DaoCloud takes care of automatically moving software hosted on GitHub (and Chinese equivalents) to the cloud. A software developer simply pushes a new version on GitHub and DaoCloud takes care of the rest: running tests,Dockerizing it, and shipping it using your preferred cloud service provider.

Several speakers were from well-recognized major software firms (I showed the conference program to non-technical people and they easily recognized several of the firm’s names). Much more so than in the US, it seems Go is not just hugely popular with newcomers and startups, but has very much found its way into larger organizations and is employed at a scale that we are only starting to see elsewhere.

Not being an expert in web services myself, in my presentation I veered off the general conference theme a bit by talking aboutgofmt and how its widespread use has started to shape expectations not just for Go but other languages as well. I presented in English but had my slides translated to Mandarin beforehand. Due to the significant language barrier I wasn’t expecting too many questions on my talk itself. Instead I decided the keep it short and leave plenty of time for general questions on Go, which the audience appreciated.

No social event in China is complete without fantastic food.

A couple of days after the conference I visited the 4-year-old startup companyQiniu (“Seven Bulls”), at the invitation of itsCEO Wei Hsu, facilitated and translated with the help of Asta Xie. Qiniu is a cloud-based storage provider for mobile applications; Wei Hsu presented at the conference and also happens to be the author of one of the first Chinese books on Go (the leftmost one in the picture above).

Qiniu lobby, engineering.

Qiniu is an extremely successful all-Go shop, with about 160 employees, serving over 150,000 companies and developers, storing over 50 Billion files, and growing by over 500 Million files per day. When asked about the reasons for Go’s success in China, Wei Hsu is quick to answer: PHP is extremely popular in China, but relatively slow and not well-suited for large systems. Like in the US, universities teach C++ and Java as primary languages, but for many applications C++ is too complex a tool and Java too bulky. In his opinion, Go now plays the role that traditionally belonged to PHP, but Go runs much faster, is type safe, and scales more easily. He loves the fact that Go is simple and applications are easy to deploy. He thought the language to be “perfect” for them and his primary request was for a recommended or even standardized package to easily access database systems. He did mention that they had GC problems in the past but were able to work around them. Hopefully our upcoming 1.5 release will address this. For Qiniu, Go appeared just at the right time and the right (open source) place.

According to Asta Xie, Qiniu is just one of many Go shops in the PRC. Large companies such as Alibaba, Baidu, Tencent, and Weibo, are now all using Go in one form or another. He pointed out that while Shanghai and neighboring cities like Suzhou are high-tech centres, even more software developers are found in the Beijing area. For 2016,  Asta hopes to organize a larger (1000, perhaps 1500 people) successor conference in Beijing.

It appears that we have found the Go users in China: They are everywhere!

Some of the GopherChina materials, including videos, are now available alongside Go coursework on a3rd party site.

Qihoo 360 and Go

$
0
0

This guest blog post was written by Yang Zhou, Software Engineer at Qihoo 360.

Qihoo 360 is a major provider of Internet and mobile security products and services in China, and operates a major Android-based mobile distribution platform. At the end of June 2014, Qihoo had about 500 million monthly active PC Internet users and over 640 million mobile users. Qihoo also operates one of China’s most popular Internet browsers and PC search engines.

My team, the Push Service Team, provides fundamental messaging services for more than 50 products across the company (both PC and mobile), including thousands of Apps in our open platform.

Our "love affair" with Go dates back to 2012 when we first attempted to provide push services for one of Qihoo’s products. The initial version was built with nginx + lua + redis, which failed to satisfy our requirement for real-time performance due to excessive load. Under these circumstances, the newly-published Go 1.0.3 release came to our attention. We completed a prototype in a matter of weeks, largely thanks to the goroutine and channel features it provided.

Initially, our Go-based system ran on 20 servers, with 20 million real-time connections in total. The system sent 2 million messages a day. That system now runs on 400 servers, supporting 200 million+ real-time connections. It now sends over 10 billion messages daily.

With rapid business expansion and increasing application needs for our push service, the initial Go system quickly reached its bottleneck: heap size went up to 69G, with maximum garbage collection (GC) pauses of 3-6 seconds. Worse still, we had to reboot the system every week to release memory. It wouldn’t be honest if we didn’t consider relinquishing Go and instead, re-writing the entire core component with C. However, things didn’t go exactly as we planned, we ran into trouble migrating the code of Business Logic Layer. As a result, it was impossible for the only personnel at that time (myself) to maintain the Go system while ensuring the logic transfer to the C service framework.

Therefore, I made the decision to stay with Go system (probably the wisest one I had to make), and great headway was made soon enough.

Here are a few tweaks we made and key take-aways:

  • Replace short connections with persistent ones (using a connection pool), to reduce creation of buffers and objects during communication.
  • Use Objects and Memory pools appropriately, to reduce the load on the GC.
  • Use a Task Pool, a mechanism with a group of long-lived goroutines consuming global task or message queues sent by connection goroutines, to replace short-lived goroutines.
  • Monitor and control goroutine numbers in the program. The lack of control can cause unbearable burden on the GC, imposed by surges in goroutines due to uninhibited acceptance of external requests, as RPC invocations sent to inner servers may block goroutines recently created.
  • Remember to add read and write deadlines to connections when under a mobile network; otherwise, it may lead to goroutine blockage. Apply it properly and with caution when under a LAN network, otherwise your RPC communication efficiency will be hurt.
  • Use Pipeline (under Full Duplex feature of TCP) to enhance the communication efficiency of RPC framework.

As a result, we successfully launched three iterations of our architecture, and two iterations of our RPC framework even with limited human resources. This can all attributed to the development convenience of Go. Below you can find the up-to-date system architecture:

The continuous improvement journey can be illustrated by a table:

Also, no temporary release of memory or system reboot is required after these optimizations.

What’s more exciting is we developed an on-line real-time Visibility Platform for profiling Go programs. We can now easily access and diagnose the system status, pinning down any potential risks. Here is a screen shot of the system in action:

The great thing about this platform is that we can actually simulate the connection and behavior of millions of online users, by applying the Distributed Stress Test Tool (also built using Go), and observe all real-time visualized data. This allows us to evaluate the effectiveness of any optimization and preclude problems by identifying system bottlenecks.

Almost every possible system optimization has been practiced so far. And we look forward to more good news from the GC team so that we could be further relieved from heavy development work. I guess our experience may also grow obsolete one day, as Go continues to evolve.

This is why I want to conclude my sharing by extending my sincere appreciation to the opportunity to attend Gopher China. It was a gala for us to learn, to share and for offering a window showcasing Go’s popularity and prosperity in China. Many other teams within Qihoo have already either got to know Go, or tried to use Go.

I am convinced that many more Chinese Internet firms will join us in re-creating their system in Go and the Go team's efforts will benefit more developers and enterprises in the foreseeable future.

Go, Open Source, Community

$
0
0

Welcome

[This is the text of my opening keynote at Gophercon 2015. We will add a link to the video when it is available.]

Thank you all for traveling to Denver to be here, and thank you to everyone watching on video. If this is your first Gophercon, welcome. If you were here last year, welcome back. Thank you to the organizers for all the work it takes to make a conference like this happen. I am thrilled to be here and to be able to talk to all of you.

I am the tech lead for the Go project and the Go team at Google. I share that role with Rob Pike. In that role, I spend a lot of time thinking about the overall Go open source project, in particular the way it runs, what it means to be open source, and the interaction between contributors inside and outside Google. Today I want to share with you how I see the Go project as a whole and then based on that explain how I see the Go open source project evolving.

Why Go?

To get started, we have to go back to the beginning. Why did we start working on Go?

Go is an attempt to make programmers more productive. We wanted to improve the software development process at Google, but the problems Google has are not unique to Google.

There were two overarching goals.

The first goal is to make a better language to meet the challenges of scalable concurrency. By scalable concurrency I mean software that deals with many concerns simultaneously, such as coordinating a thousand back end servers by sending network traffic back and forth.

Today, that kind of software has a shorter name: we call it cloud software. It's fair to say that Go was designed for the cloud before clouds ran software.

The larger goal is to make a better environment to meet the challenges of scalable software development, software worked on and used by many people, with limited coordination between them, and maintained for years. At Google we have thousands of engineers writing and sharing their code with each other, trying to get their work done, reusing the work of others as much as possible, and working in a code base with a history dating back over ten years. Engineers often work on or at least look at code originally written by someone else, or that they wrote years ago, which often amounts to the same thing.

That situation inside Google has a lot in common with large scale, modern open source development as practiced on sites like GitHub. Because of this, Go is a great fit for open source projects, helping them accept and manage contributions from a large community over a long period of time.

I believe much of Go's success is explained by the fact that Go is a great fit for cloud software, Go is a great fit for open source projects, and, serendipitously, both of those are growing in popularity and importance in the software industry.

Other people have made similar observations. Here are two. Last year, on RedMonk.com, Donnie Berkholz wrote about “Go as the emerging language of cloud infrastructure,” observing that“[Go's] marquee projects ... are cloud-centric or otherwise made for dealing with distributed systems or transient environments.”

This year, on Texlution.com, the author wrote an article titled“Why Golang is doomed to succeed,” pointing out that this focus on large-scale development was possibly even better suited to open source than to Google itself: “This open source fitness is why I think you are about to see more and more Go around ...”

The Go Balance

How does Go accomplish those things?

How does it make scalable concurrency and scalable software development easier?

Most people answer this question by talking about channels and goroutines, and interfaces, and fast builds, and the go command, and good tool support. Those are all important parts of the answer, but I think there is a broader idea behind them.

I think of that idea as Go's balance. There are competing concerns in any software design, and there is a very natural tendency to try to solve all the problems you foresee. In Go, we have explicitly tried not to solve everything. Instead, we've tried to do just enough that you can build your own custom solutions easily.

The way I would summarize Go's chosen balance is this: Do Less. Enable More.

Do less, but enable more.

Go can't do everything. We shouldn't try. But if we work at it, Go can probably do a few things well. If we select those things carefully, we can lay a foundation on which developers can easily build the solutions and tools they need, and ideally can interoperate with the solutions and tools built by others.

Examples

Let me illustrate this with some examples.

First, the size of the Go language itself. We worked hard to put in as few concepts as possible, to avoid the problem of mutually incomprehensible dialects forming in different parts of a large developer community. No idea went into Go until it had been simplified to its essence and then had clear benefits that justified the complexity being added.

In general, if we have 100 things we want Go to do well, we can't make 100 separate changes. Instead, we try to research and understand the design space and then identify a few changes that work well together and that enable maybe 90 of those things. We're willing to sacrifice the remaining 10 to avoid bloating the language, to avoid adding complexity only to address specific use cases that seem important today but might be gone tomorrow.

Keeping the language small enables more important goals. Being small makes Go easier to learn, easier to understand, easier to implement, easier to reimplement, easier to debug, easier to adjust, and easier to evolve. Doing less enables more.

I should point out that this means we say no to a lot of other people's ideas, but I assure you we've said no to even more of our own ideas.

Next, channels and goroutines. How should we structure and coordinate concurrent and parallel computations? Mutexes and condition variables are very general but so low-level that they're difficult to use correctly. Parallel execution frameworks like OpenMP are so high-level that they can only be used to solve a narrow range of problems. Channels and goroutines sit between these two extremes. By themselves, they aren't a solution to much. But they are powerful enough to be easily arranged to enable solutions to many common problems in concurrent software. Doing less—really doing just enough—enables more.

Next, types and interfaces. Having static types enables useful compile-time checking, something lacking in dynamically-typed languages like Python or Ruby. At the same time, Go's static typing avoids much of the repetition of traditional statically typed languages, making it feel more lightweight, more like the dynamically-typed languages. This was one of the first things people noticed, and many of Go's early adopters came from dynamically-typed languages.

Go's interfaces are a key part of that. In particular, omitting the ``implements'' declarations of Java or other languages with static hierarchy makes interfaces lighter weight and more flexible. Not having that rigid hierarchy enables idioms such as test interfaces that describe existing, unrelated production implementations. Doing less enables more.

Next, testing and benchmarking. Is there any shortage of testing and benchmarking frameworks in most languages? Is there any agreement between them?

Go's testing package is not meant to address every possible facet of these topics. Instead, it is meant to provide the basic concepts necessary for most higher-level tooling. Packages have test cases that pass, fail, or are skipped. Packages have benchmarks that run and can be measured by various metrics.

Doing less here is an attempt to reduce these concepts to their essence, to create a shared vocabulary so that richer tools can interoperate. That agreement enables higher-level testing software like Miki Tebeka's go2xunit converter, or the benchcmp and benchstat benchmark analysis tools.

Because there is agreement about the representation of the basic concepts, these higher-level tools work for all Go packages, not just ones that make the effort to opt in, and they interoperate with each other, in that using, say, go2xunit does not preclude also using benchstat, the way it would if these tools were, say, plugins for competing testing frameworks. Doing less enables more.

Next, refactoring and program analysis. Because Go is for large code bases, we knew it would need to support automatic maintenance and updating of source code. We also knew that this topic was too large to build in directly. But we knew one thing that we had to do. In our experience attempting automated program changes in other settings, the most significant barrier we hit was actually writing the modified program out in a format that developers can accept.

In other languages, it's common for different teams to use different formatting conventions. If an edit by a program uses the wrong convention, it either writes a section of the source file that looks nothing like the rest of the file, or it reformats the entire file, causing unnecessary and unwanted diffs.

Go does not have this problem. We designed the language to make gofmt possible, we worked hard to make gofmt's formatting acceptable for all Go programs, and we made sure gofmt was there from day one of the original public release. Gofmt imposes such uniformity that automated changes blend into the rest of the file. You can't tell whether a particular change was made by a person or a computer. We didn't build explicit refactoring support. Establishing an agreed-upon formatting algorithm was enough of a shared base for independent tools to develop and to interoperate. Gofmt enabled gofix, goimports, eg, and other tools. I believe the work here is only just getting started. Even more can be done.

Last, building and sharing software. In the run up to Go 1, we built goinstall, which became what we all know as "go get". That tool defined a standard zero-configuration way to resolve import paths on sites like github.com, and later a way to resolve paths on other sites by making HTTP requests. This agreed-upon resolution algorithm enabled other tools that work in terms of those paths, most notably Gary Burd's creation of godoc.org. In case you haven't used it, you go to godoc.org/the-import-path for any valid "go get" import path, and the web site will fetch the code and show you the documentation for it. A nice side effect of this has been that godoc.org serves as a rough master list of the Go packages publicly available. All we did was give import paths a clear meaning. Do less, enable more.

You'll notice that many of these tooling examples are about establishing a shared convention. Sometimes people refer to this as Go being “opinionated,” but there's something deeper going on. Agreeing to the limitations of a shared convention is a way to enable a broad class of tools that interoperate, because they all speak the same base language. This is a very effective way to do less but enable more. Specifically, in many cases we can do the minimum required to establish a shared understanding of a particular concept, like remote imports, or the proper formatting of a source file, and thereby enable the creation of packages and tools that work together because they all agree about those core details.

I'm going to return to that idea later.

Why is Go open source?

But first, as I said earlier, I want to explain how I see the balance of Do Less and Enable More guiding our work on the broader Go open source project. To do that, I need to start with why Go is open source at all.

Google pays me and others to work on Go, because, if Google's programmers are more productive, Google can build products faster, maintain them more easily, and so on. But why open source Go? Why should Google share this benefit with the world?

Of course, many of us worked on open source projects before Go, and we naturally wanted Go to be part of that open source world. But our preferences are not a business justification. The business justification is that Go is open source because that's the only way that Go can succeed. We, the team that built Go within Google, knew this from day one. We knew that Go had to be made available to as many people as possible for it to succeed.

Closed languages die.

A language needs large, broad communities.

A language needs lots of people writing lots of software, so that when you need a particular tool or library, there's a good chance it has already been written, by someone who knows the topic better than you, and who spent more time than you have to make it great.

A language needs lots of people reporting bugs, so that problems are identified and fixed quickly. Because of the much larger user base, the Go compilers are much more robust and spec-compliant than the Plan 9 C compilers they're loosely based on ever were.

A language needs lots of people using it for lots of different purposes, so that the language doesn't overfit to one use case and end up useless when the technology landscape changes.

A language needs lots of people who want to learn it, so that there is a market for people to write books or teach courses, or run conferences like this one.

None of this could have happened if Go had stayed within Google. Go would have suffocated inside Google, or inside any single company or closed environment.

Fundamentally, Go must be open, and Go needs you. Go can't succeed without all of you, without all the people using Go for all different kinds of projects all over the world.

In turn, the Go team at Google could never be large enough to support the entire Go community. To keep scaling, we need to enable all this ``more'' while doing less. Open source is a huge part of that.

Go's open source

What does open source mean? The minimum requirement is to open the source code, making it available under an open source license, and we've done that.

But we also opened our development process: since announcing Go, we've done all our development in public, on public mailing lists open to all. We accept and review source code contributions from anyone. The process is the same whether you work for Google or not. We maintain our bug tracker in public, we discuss and develop proposals for changes in public, and we work toward releases in public. The public source tree is the authoritative copy. Changes happen there first. They are only brought into Google's internal source tree later. For Go, being open source means that this is a collective effort that extends beyond Google, open to all.

Any open source project starts with a few people, often just one, but with Go it was three: Robert Griesemer, Rob Pike, and Ken Thompson. They had a vision of what they wanted Go to be, what they thought Go could do better than existing languages, and Robert will talk more about that tomorrow morning. I was the next person to join the team, and then Ian Taylor, and then, one by one, we've ended up where we are today, with hundreds of contributors.

Thank You to the many people who have contributed code or ideas or bug reports to the Go project so far. We tried to list everyone we could in our space in the program today. If your name is not there, I apologize, but thank you.

I believe the hundreds of contributors so far are working toward a shared vision of what Go can be. It's hard to put words to these things, but I did my best to explain one part of the vision earlier: Do Less, Enable More.

Google's role

A natural question is: What is the role of the Go team at Google, compared to other contributors? I believe that role has changed over time, and it continues to change. The general trend is that over time the Go team at Google should be doing less and enabling more.

In the very early days, before Go was known to the public, the Go team at Google was obviously working by itself. We wrote the first draft of everything: the specification, the compiler, the runtime, the standard library.

Once Go was open sourced, though, our role began to change. The most important thing we needed to do was communicate our vision for Go. That's difficult, and we're still working at it.. The initial implementation was an important way to communicate that vision, as was the development work we led that resulted in Go 1, and the various blog posts, and articles, and talks we've published.

But as Rob said at Gophercon last year,"the language is done." Now we need to see how it works, to see how people use it, to see what people build. The focus now is on expanding the kind of work that Go can help with.

Google's primarily role is now to enable the community, to coordinate, to make sure changes work well together, and to keep Go true to the original vision.

Google's primary role is: Do Less. Enable More.

I mentioned earlier that we'd rather have a small number of features that enable, say, 90% of the target use cases, and avoid the orders of magnitude more features necessary to reach 99 or 100%. We've been successful in applying that strategy to the areas of software that we know well. But if Go is to become useful in many new domains, we need experts in those areas to bring their expertise to our discussions, so that together we can design small adjustments that enable many new applications for Go.

This shift applies not just to design but also to development. The role of the Go team at Google continues to shift more to one of guidance and less of pure development. I certainly spend much more time doing code reviews than writing code, more time processing bug reports than filing bug reports myself. We need to do less and enable more.

As design and development shift to the broader Go community, one of the most important things we the original authors of Go can offer is consistency of vision, to help keep Go Go. The balance that we must strike is certainly subjective. For example, a mechanism for extensible syntax would be a way to enable more ways to write Go code, but that would run counter to our goal of having a consistent language without different dialects.

We have to say no sometimes, perhaps more than in other language communities, but when we do, we aim to do so constructively and respectfully, to take that as an opportunity to clarify the vision for Go.

Of course, it's not all coordination and vision. Google still funds Go development work. Rick Hudson is going to talk later today about his work on reducing garbage collector latency, and Hana Kim is going to talk tomorrow about her work on bringing Go to mobile devices. But I want to make clear that, as much as possible, we aim to treat development funded by Google as equal to development funded by other companies or contributed by individuals using their spare time. We do this because we don't know where the next great idea will come from. Everyone contributing to Go should have the opportunity to be heard.

Examples

I want to share some evidence for this claim that, over time, the original Go team at Google is focusing more on coordination than direct development.

First, the sources of funding for Go development are expanding. Before the open source release, obviously Google paid for all Go development. After the open source release, many individuals started contributing their time, and we've slowly but steadily been growing the number of contributors supported by other companies to work on Go at least part-time, especially as it relates to making Go more useful for those companies. Today, that list includes Canonical, Dropbox, Intel, Oracle, and others. And of course Gophercon and the other regional Go conferences are organized entirely by people outside Google, and they have many corporate sponsors besides Google.

Second, the conceptual depth of Go development done outside the original team is expanding.

Immediately after the open source release, one of the first large contributions was the port to Microsoft Windows, started by Hector Chu and completed by Alex Brainman and others. More contributors ported Go to other operating systems. Even more contributors rewrote most of our numeric code to be faster or more precise or both. These were all important contributions, and very much appreciated, but for the most part they did not involve new designs.

More recently, a group of contributors led by Aram Hăvărneanu ported Go to the ARM 64 architecture, This was the first architecture port by contributors outside Google. This is significant, because in general support for a new architecture requires more design work than support for a new operating system. There is more variation between architectures than between operating systems.

Another example is the introduction over the past few releases of preliminary support for building Go programs using shared libraries. This feature is important for many Linux distributions but not as important for Google, because we deploy static binaries. We have been helping guide the overall strategy, but most of the design and nearly all of the implementation has been done by contributors outside Google, especially Michael Hudson-Doyle.

My last example is the go command's approach to vendoring. I define vendoring as copying source code for external dependencies into your tree to make sure that they doesn't disappear or change underfoot.

Vendoring is not a problem Google suffers, at least not the way the rest of the world does. We copy open source libraries we want to use into our shared source tree, record what version we copied, and only update the copy when there is a need to do so. We have a rule that there can only be one version of a particular library in the source tree, and it's the job of whoever wants to upgrade that library to make sure it keeps working as expected by the Google code that depends on it. None of this happens often. This is the lazy approach to vendoring.

In contrast, most projects outside Google take a more eager approach, importing and updating code using automated tools and making sure that they are always using the latest versions.

Because Google has relatively little experience with this vendoring problem, we left it to users outside Google to develop solutions. Over the past five years, people have built a series of tools. The main ones in use today are Keith Rarick's godep, Owen Ou's nut, and the gb-vendor plugin for Dave Cheney's gb,

There are two problems with the current situation. The first is that these tools are not compatible out of the box with the go command's "go get". The second is that the tools are not even compatible with each other. Both of these problems fragment the developer community by tool.

Last fall, we started a public design discussion to try to build consensus on some basics about how these tools all operate, so that they can work alongside "go get" and each other.

Our basic proposal was that all tools agree on the approach of rewriting import paths during vendoring, to fit with "go get"'s model, and also that all tools agree on a file format describing the source and version of the copied code, so that the different vendoring tools can be used together even by a single project. If you use one today, you should still be able to use another tomorrow.

Finding common ground in this way was very much in the spirit of Do Less, Enable More. If we could build consensus about these basic semantic aspects, that would enable "go get" and all these tools to interoperate, and it would enable switching between tools, the same way that agreement about how Go programs are stored in text files enables the Go compiler and all text editors to interoperate. So we sent out our proposal for common ground.

Two things happened.

First, Daniel Theophanes started a vendor-spec project on GitHub with a new proposal and took over coordination and design of the spec for vendoring metadata.

Second, the community spoke with essentially one voice to say that rewriting import paths during vendoring was not tenable. Vendoring works much more smoothly if code can be copied without changes.

Keith Rarick posted an alternate proposal for a minimal change to the go command to support vendoring without rewriting import paths. Keith's proposal was configuration-free and fit in well with the rest of the go command's approach. That proposal will ship as an experimental feature in Go 1.5 and likely enabled by default in Go 1.6. And I believe that the various vendoring tool authors have agreed to adopt Daniel's spec once it is finalized.

The result is that at the next Gophercon we should have broad interoperability between vendoring tools and the go command, and the design to make that happen was done entirely by contributors outside the original Go team.

Not only that, the Go team's proposal for how to do this was essentially completely wrong. The Go community told us that very clearly. We took that advice, and now there's a plan for vendoring support that I believe everyone involved is happy with.

This is also a good example of our general approach to design. We try not to make any changes to Go until we feel there is broad consensus on a well-understood solution. For vendoring, feedback and design from the Go community was critical to reaching that point.

This general trend toward both code and design coming from the broader Go community is important for Go. You, the broader Go community, know what is working and what is not in the environments where you use Go. We at Google don't. More and more, we will rely on your expertise, and we will try to help you develop designs and code that extend Go to be useful in more settings and fit well with Go's original vision. At the same time, we will continue to wait for broad consensus on well-understood solutions.

This brings me to my last point.

Code of Conduct

I've argued that Go must be open, and that Go needs your help.

But in fact Go needs everyone's help. And everyone isn't here.

Go needs ideas from as many people as possible.

To make that a reality, the Go community needs to be as inclusive, welcoming, helpful, and respectful as possible.

The Go community is large enough now that, instead of assuming that everyone involved knows what is expected, I and others believe that it makes sense to write down those expectations explicitly. Much like the Go spec sets expectations for all Go compilers, we can write a spec setting expectations for our behavior in online discussions and in offline meetings like this one.

Like any good spec, it must be general enough to allow many implementations but specific enough that it can identify important problems. When our behavior doesn't meet the spec, people can point that out to us, and we can fix the problem. At the same time, it's important to understand that this kind of spec cannot be as precise as a language spec. We must start with the assumption that we will all be reasonable in applying it.

This kind of spec is often referred to as a Code of Conduct. Gophercon has one, which we've all agreed to follow by being here, but the Go community does not. I and others believe the Go community needs a Code of Conduct.

But what should it say?

I believe the most important overall statement we can make is that if you want to use or discuss Go, then you are welcome here, in our community. That is the standard I believe we aspire to.

If for no other reason (and, to be clear, there are excellent other reasons), Go needs as large a community as possible. To the extent that behavior limits the size of the community, it holds Go back. And behavior can easily limit the size of the community.

The tech community in general and the Go community in particular is skewed toward people who communicate bluntly. I don't believe this is fundamental. I don't believe this is necessary. But it's especially easy to do in online discussions like email and IRC, where plain text is not supplemented by the other cues and signals we have in face-to-face interactions.

For example, I have learned that when I am pressed for time I tend to write fewer words, with the end result that my emails seem not just hurried but blunt, impatient, even dismissive. That's not how I feel, but it's how I can come across, and that impression can be enough to make people think twice about using or contributing to Go. I realized I was doing this when some Go contributors sent me private email to let me know. Now, when I am pressed for time, I pay extra attention to what I'm writing, and I often write more than I naturally would, to make sure I'm sending the message I intend.

I believe that correcting the parts of our everyday interactions, intended or not, that drive away potential users and contributors is one of the most important things we can all do to make sure the Go community continues to grow. A good Code of Conduct can help us do that.

We have no experience writing a Code of Conduct, so we have been reading existing ones, and we will probably adopt an existing one, perhaps with minor adjustments. The one I like the most is the Django Code of Conduct, which originated with another project called SpeakUp! It is structured as an elaboration of a list of reminders for everyday interaction.

"Be friendly and patient. Be welcoming. Be considerate. Be respectful. Be careful in the words that you choose. When we disagree, try to understand why."

I believe this captures the tone we want to set, the message we want to send, the environment we want to create for new contributors. I certainly want to be friendly, patient, welcoming, considerate, and respectful. I won't get it exactly right all the time, and I would welcome a helpful note if I'm not living up to that. I believe most of us feel the same way.

I haven't mentioned active exclusion based on or disproportionately affecting race, gender, disability, or other personal characteristics, and I haven't mentioned harassment. For me, it follows from what I just said that exclusionary behavior or explicit harassment is absolutely unacceptable, online and offline. Every Code of Conduct says this explicitly, and I expect that ours will too. But I believe the SpeakUp! reminders about everyday interactions are an equally important statement. I believe that setting a high standard for those everyday interactions makes extreme behavior that much clearer and easier to deal with.

I have no doubts that the Go community can be one of the most friendly, welcoming, considerate, and respectful communities in the tech industry. We can make that happen, and it will be a benefit and credit to us all.

Andrew Gerrand has been leading the effort to adopt an appropriate Code of Conduct for the Go community. If you have suggestions, or concerns, or experience with Codes of Conduct, or want to be involved, please find Andrew or me during the conference. If you'll still be here on Friday, Andrew and I are going to block off some time for Code of Conduct discussions during Hack Day.

Again, we don't know where the next great idea will come from. We need all the help we can get. We need a large, diverse Go community.

Thank You

I consider the many people releasing software for download using “go get,” sharing their insights via blog posts, or helping others on the mailing lists or IRC to be part of this broad open source effort, part of the Go community. Everyone here today is also part of that community.

Thank you in advance to the presenters who over the next few days will take time to share their experiences using and extending Go.

Thank you in advance to all of you in the audience for taking the time to be here, to ask questions, and to let us know how Go is working for you. When you go back home, please continue to share what you've learned. Even if you don't use Go for daily work, we'd love to see what's working for Go adopted in other contexts, just as we're always looking for good ideas to bring back into Go.

Thank you all again for making the effort to be here and for being part of the Go community.

For the next few days, please: tell us what we're doing right, tell us what we're doing wrong, and help us all work together to make Go even better.

Remember to be friendly, patient, welcoming, considerate, and respectful.

Above all, enjoy the conference.


GopherCon 2015 Roundup

$
0
0

A few weeks ago, Go programmers from around the world descended on Denver, Colorado for GopherCon 2015. The two-day, single-track conference attracted more than 1,250 attendees—nearly double last year's number—and featured 22 talks presented by Go community members.

The Cowboy Gopher (a toy given to each attendee) watches over the ranch.
Photograph by Nathan Youngman. Gopher by Renee French.

Today the organizers have posted the videos online so you can now enjoy the conference from afar:

Day 1:

  • Go, Open Source, Community — Russ Cox (video) (text)
  • Go kit: A Standard Library for Distributed Programming — Peter Bourgon (video) (slides)
  • Delve Into Go — Derek Parker (video) (slides)
  • How a complete beginner learned Go as her first backend language in 5 weeks — Audrey Lim (video) (slides)
  • A Practical Guide to Preventing Deadlocks and Leaks in Go — Richard Fliam (video)
  • Go GC: Solving the Latency Problem — Rick Hudson (video) (slides)
  • Simplicity and Go — Katherine Cox-Buday (video) (slides)
  • Rebuilding Parse.com in Go - an opinionated rewrite — Abhishek Kona (video) (slides)
  • Prometheus: Designing and Implementing a Modern Monitoring Solution in Go — Björn Rabenstein (video) (slides)
  • What Could Go Wrong? — Kevin Cantwell (video)
  • The Roots of Go — Baishampayan Ghose (video) (slides)

Day 2:

  • The Evolution of Go — Robert Griesemer (video) (slides)
  • Static Code Analysis Using SSA — Ben Johnson (video) (slides)
  • Go on Mobile — Hana Kim (video) (slides)
  • Go Dynamic Tools — Dmitry Vyukov (video) (slides)
  • Embrace the Interface — Tomás Senart (video) (slides)
  • Uptime: Building Resilient Services with Go — Blake Caldwell (video) (slides)
  • Cayley: Building a Graph Database — Barak Michener (video) (slides)
  • Code Generation For The Sake Of Consistency — Sarah Adams (video)
  • The Many Faces of Struct Tags — Sam Helman and Kyle Erf (video) (slides)
  • Betting the Company on Go and Winning — Kelsey Hightower (video)
  • How Go Was Made — Andrew Gerrand (video) (slides)

The hack day was also a ton of fun, with hours of lightning talks and a range of activities from programming robots to a Magic: the Gathering tournament.

Huge thanks to the event organizers Brian Ketelsen and Eric St. Martin and their production team, the sponsors, the speakers, and the attendees for making this such a fun and action-packed conference. Hope to see you there next year!

Go 1.5 is released

$
0
0

Today the Go project is proud to release Go 1.5, the sixth major stable release of Go.

This release includes significant changes to the implementation. The compiler tool chain was translated from C to Go, removing the last vestiges of C code from the Go code base. The garbage collector was completely redesigned, yielding a dramatic reduction in garbage collection pause times. Related improvements to the scheduler allowed us to change the defaultGOMAXPROCS value (the number of concurrently executing goroutines) from 1 to the number of logical CPUs. Changes to the linker enable distributing Go packages as shared libraries to link into Go programs, and building Go packages into archives or shared libraries that may be linked into or loaded by C programs (design doc).

The release also includes improvements to the developer tools. Support for "internal" packages permits sharing implementation details between packages.Experimental support for "vendoring" external dependencies is a step toward a standard mechanism for managing dependencies in Go programs. The new "go tool trace" command enables the visualisation of  program traces generated by new tracing infrastructure in the runtime. The new "go doc" command provides an improved command-line interface for viewing Go package documentation.

There are also several new operating system and architecture ports. The more mature new ports are darwin/arm, darwin/arm64 (Apple's iPhone and iPad devices), and linux/arm64. There is also experimental support for ppc64 and ppc64le (IBM 64-bit PowerPC, big and little endian).

The new darwin/arm64 port and external linking features fuel theGo mobile project, an experiment to see how Go might be used for building apps on Android and iOS devices. (The Go mobile work itself is not part of this release.)

The only language change is very minor,the lifting of a restriction in the map literal syntax to make them more succinct and consistent with slice literals.

The standard library saw many additions and improvements, too. The flag package now shows cleaner usage messages. The math/big package now provides a Float type for computing with arbitrary-precision floating point numbers. An improvement to the DNS resolver on Linux and BSD systems has removed the cgo requirement for programs that do name lookups. The go/types package has beenmoved to the standard library from the golang.org/x/tools repository. (The new go/constant andgo/importer packages are also a result of this move.) The reflect package has added theArrayOf andFuncOf functions, analogous to the existing SliceOf function. And, of course, there is the usuallist of smaller fixes and improvements.

For the full story, see the detailed release notes. Or if you just can't wait to get started, head over to the downloads page to get Go 1.5 now.

Go GC: Prioritizing low latency and simplicity

$
0
0

The Setup

Go is building a garbage collector (GC) not only for 2015 but for 2025 and beyond: A GC that supports today’s software development and scales along with new software and hardware throughout the next decade. Such a future has no place for stop-the-world GC pauses, which have been an impediment to broader uses of safe and secure languages such as Go.

Go 1.5, the first glimpse of this future, achieves GC latencies well below the 10 millisecond goal we set a year ago. We presented some impressive numbers in a talk at Gophercon. The latency improvements have generated a lot of attention; Robin Verlangen’s blog postBillions of requests per day meet Go 1.5 validates our direction with end to end results. We also particularly enjoyedAlan Shreve’s production server graphs and his "Holy 85% reduction" comment.

Today 16 gigabytes of RAM costs $100 and CPUs come with many cores, each with multiple hardware threads. In a decade this hardware will seem quaint but the software being built in Go today will need to scale to meet expanding needs and the next big thing. Given that hardware will provide the power to increase throughput, Go’s garbage collector is being designed to favor low latency and tuning via only a single knob. Go 1.5 is the first big step down this path and these first steps will forever influence Go and the applications it best supports. This blog post gives a high-level overview of what we have done for the Go 1.5 collector.

The Embellishment

To create a garbage collector for the next decade, we turned to an algorithm from decades ago. Go's new garbage collector is a concurrent, tri-color,mark-sweep collector, an idea first proposed byDijkstra in 1978. This is a deliberate divergence from most "enterprise" grade garbage collectors of today, and one that we believe is well suited to the properties of modern hardware and the latency requirements of modern software.

In a tri-color collector, every object is either white, grey, or black and we view the heap as a graph of connected objects. At the start of a GC cycle all objects are white. The GC visits all roots, which are objects directly accessible by the application such as globals and things on the stack, and colors these grey. The GC then chooses a grey object, blackens it, and then scans it for pointers to other objects. When this scan finds a pointer to a white object, it turns that object grey. This process repeats until there are no more grey objects. At this point, white objects are known to be unreachable and can be reused.

This all happens concurrently with the application, known as the mutator, changing pointers while the collector is running. Hence, the mutator must maintain the invariant that no black object points to a white object, lest the garbage collector lose track of an object installed in a part of the heap it has already visited. Maintaining this invariant is the job of thewrite barrier, which is a small function run by the mutator whenever a pointer in the heap is modified. Go’s write barrier colors the now-reachable object grey if it is currently white, ensuring that the garbage collector will eventually scan it for pointers.

Deciding when the job of finding all grey objects is done is subtle and can be expensive and complicated if we want to avoid blocking the mutators. To keep things simple Go 1.5 does as much work as it can concurrently and then briefly stops the world to inspect all potential sources of grey objects. Finding the sweet spot between the time needed for this final stop-the-world and the total amount of work that this GC does is a major deliverable for Go 1.6.

Of course the devil is in the details. When do we start a GC cycle? What metrics do we use to make that decision? How should the GC interact with the Go scheduler? How do we pause a mutator thread long enough to scan its stack? How do we represent white, grey, and black so we can efficiently find and scan grey objects? How do we know where the roots are? How do we know where in an object pointers are located? How do we minimize memory fragmentation? How do we deal with cache performance issues? How big should the heap be? And on and on, some related to allocation, some to finding reachable objects, some related to scheduling, but many related to performance. Low-level discussions of each of these areas are beyond the scope of this blog post.

At a higher level, one approach to solving performance problems is to add GC knobs, one for each performance issue. The programmer can then turn the knobs in search of appropriate settings for their application. The downside is that after a decade with one or two new knobs each year you end up with the GC Knobs Turner Employment Act. Go is not going down that path. Instead we provide a single knob, called GOGC. This value controls the total size of the heap relative to the size of reachable objects. The default value of 100 means that total heap size is now 100% bigger than (i.e., twice) the size of the reachable objects after the last collection. 200 means total heap size is 200% bigger than (i.e., three times) the size of the reachable objects. If you want to lower the total time spent in GC, increase GOGC. If you want to trade more GC time for less memory, lower GOGC.

More importantly as RAM doubles with the next generation of hardware, simply doubling GOGC will halve the number of GC cycles. On the other hand since GOGC is based on reachable object size, doubling the load by doubling the reachable objects requires no retuning. The application just scales. Furthermore, unencumbered by ongoing support for dozens of knobs, the runtime team can focus on improving the runtime based on feedback from real customer applications.

The Punchline

Go 1.5’s GC ushers in a future where stop-the-world pauses are no longer a barrier to moving to a safe and secure language. It is a future where applications scale effortlessly along with hardware and as hardware becomes more powerful the GC will not be an impediment to better, more scalable software. It’s a good place to be for the next decade and beyond. For more details about the 1.5 GC and how we eliminated latency issues see theGo GC: Latency Problem Solved presentation or the slides.

GolangUK 2015

$
0
0

On August 21st the Go community gathered in London for the first edition ofGolang UK. The conference featured two parallel tracks and nearly 400 gophers attended.

The conference started with the opening keynote by David Calavera called Crossing the Language Chasm (video) and continued with two concurrently executed tracks.

Main track:

Side track:

Finally Damian Gryski took the stage for the closing keynote (video), giving an overview of how the Go community has evolved over time and hinting to what the future might look like.

On the day before the conference William Kennedy gave a full day Go workshop.

It was a great conference, so congratulations to the organizers and see you next year in London!

Six years of Go

$
0
0

Six years ago today the Go language was released as an open source project. Since then, more than 780 contributors have made over 30,000 commits to the project's 22 repositories. The ecosystem continues to grow, with GitHub reporting more than 90,000 Go repositories. And, offline, we see new Go events and user groups pop up aroundtheworld with regularity.

In August we released Go 1.5, the most significant release since Go 1. It features a completelyredesigned garbage collector that makes the language more suitable for latency-sensitive applications; it marks the transition from a C-based compiler tool chain to onewritten entirely in Go; and it includes ports to new architectures, with better support for ARM processors (the chips that power most smartphones). These improvements make Go better suited to a broader range of tasks, a trend that we hope will continue over the coming years.

Improvements to tools continue to boost developer productivity. We introduced the execution tracer and the"go doc" command, as well as more enhancements to our variousstatic analysis tools. We are also working on anofficial Go plugin for Sublime Text, with better support for other editors in the pipeline.

Early next year we will release more improvements in Go 1.6, including HTTP/2 support for net/http servers and clients, an official package vendoring mechanism, support for blocks in text and HTML templates, a memory sanitizer that checks both Go and C/C++ code, and the usual assortment of other improvements and fixes.

This is the sixth time we have had the pleasure of writing a birthday blog post for Go, and we would not be doing so if not for the wonderful and passionate people in our community. The Go team would like to thank everyone who has contributed code, written an open source library, authored a blog post, helped a new gopher, or just given Go a try. Without you, Go would not be as complete, useful, or successful as it is today. Thank you, and celebrate!

Language and Locale Matching in Go

$
0
0

Introduction

Consider an application, such as a web site, with support for multiple languages in its user interface. When a user arrives with a list of preferred languages, the application must decide which language it should use in its presentation to the user. This requires finding the best match between the languages the application supports and those the user prefers. This post explains why this is a difficult decision and how Go can help.

Language Tags

Language tags, also known as locale identifiers, are machine-readable identifiers for the language and/or dialect being used. The most common reference for them is the IETF BCP 47 standard, and that is the standard the Go libraries follow. Here are some examples of BCP 47 language tags and the language or dialect they represent.

TagDescription
enEnglish
en-USAmerican English
cmnMandarin Chinese
zhChinese, typically Mandarin
nlDutch
nl-BEFlemish
es-419Latin American Spanish
az, az-Latnboth Azerbaijani written in Latin script
az-ArabAzerbaijani written in Arabic

The general form of the language tag is a language code (“en”, “cmn”, “zh”, “nl”, “az” above) followed by an optional subtag for script (“-Arab”), region (“-US”, “-BE”, “-419”), variants (“-oxendict” for Oxford English Dictionary spelling), and extensions (“-u-co-phonebk” for phone-book sorting). The most common form is assumed if a subtag is omitted, for instance“az-Latn-AZ” for “az”.

The most common use of language tags is to select from a set of system-supported languages according to a list of the user's language preferences, for example deciding that a user who prefers Afrikaans would be best served (assuming Afrikaans is not available) by the system showing Dutch. Resolving such matches involves consulting data on mutual language comprehensibility.

The tag resulting from this match is subsequently used to obtain language-specific resources such as translations, sorting order, and casing algorithms. This involves a different kind of matching. For example, as there is no specific sorting order for Portuguese, a collate package may fall back to the sorting order for the default, or “root”, language.

The Messy Nature of Matching Languages

Handling language tags is tricky. This is partly because the boundaries of human languages are not well defined and partly because of the legacy of evolving language tag standards. In this section we will show some of the messy aspects of handling language tags.

Tags with different language codes can indicate the same language

For historical and political reasons, many language codes have changed over time, leaving languages with an older legacy code as well as a new one. But even two current codes may refer to the same language. For example, the official language code for Mandarin is “cmn”, but “zh” is by far the most commonly used designator for this language. The code “zh” is officially reserved for a so called macro language, identifying the group of Chinese languages. Tags for macro languages are often used interchangeably with the most-spoken language in the group.

Matching language code alone is not sufficient

Azerbaijani (“az”), for example, is written in different scripts depending on the country in which it is spoken: "az-Latn" for Latin (the default script),"az-Arab" for Arabic, and “az-Cyrl” for Cyrillic. If you replace "az-Arab" with just "az", the result will be in Latin script and may not be understandable to a user who only knows the Arabic form.

Also different regions may imply different scripts. For example: “zh-TW” and “zh-SG” respectively imply the use of Traditional and Simplified Han. As another example, “sr” (Serbian) defaults to Cyrillic script, but “sr-RU” (Serbian as written in Russia) implies the Latin script! A similar thing can be said for Kyrgyz and other languages.

If you ignore subtags, you might as well present Greek to the user.

The best match might be a language not listed by the user

The most common written form of Norwegian (“nb”) looks an awful lot like Danish. If Norwegian is not available, Danish may be a good second choice. Similarly, a user requesting Swiss German (“gsw”) will likely be happy to be presented German (“de”), though the converse is far from true. A user requesting Uygur may be happier to fall back to Chinese than to English. Other examples abound. If a user-requested language is not supported, falling back to English is often not the best thing to do.

The choice of language decides more than translation

Suppose a user asks for Danish, with German as a second choice. If an application chooses German, it must not only use German translations but also use German (not Danish) collation. Otherwise, for example, a list of animals might sort “Bär” before “Äffin”.

Selecting a supported language given the user’s preferred languages is like a handshaking algorithm: first you determine which protocol to communicate in (the language) and then you stick with this protocol for all communication for the duration of a session.

Using a “parent” of a language as fallback is non-trivial

Suppose your application supports Angolan Portuguese (“pt-AO”). Packages in golang.org/x/text, like collation and display, may not have specific support for this dialect. The correct course of action in such cases is to match the closest parent dialect. Languages are arranged in a hierarchy, with each specific language having a more general parent. For example, the parent of “en-GB-oxendict” is “en-GB”, whose parent is “en”, whose parent is the undefined language “und”, also known as the root language. In the case of collation, there is no specific collation order for Portugese, so the collate package will select the sorting order of the root language. The closest parent to Angolan Portuguese supported by the display package is European Portuguese (“pt-PT”) and not the more obvious “pt”, which implies Brazilian.

In general, parent relationships are non-trivial. To give a few more examples, the parent of “es-CL” is “es-419”, the parent of“zh-TW” is “zh-Hant”, and the parent of “zh-Hant” is “und”. If you compute the parent by simply removing subtags, you may select a “dialect” that is incomprehensible to the user.

Language Matching in Go

The Go package golang.org/x/text/language implements the BCP 47 standard for language tags and adds support for deciding which language to use based on data published in the Unicode Common Locale Data Repository (CLDR).

Here is a sample program, explained below, matching a user's language preferences against an application's supported languages:

package mainimport ("fmt""golang.org/x/text/language""golang.org/x/text/language/display")var userPrefs = []language.Tag{    language.Make("gsw"), // Swiss German    language.Make("fr"),  // French}var serverLangs = []language.Tag{    language.AmericanEnglish, // en-US fallback    language.German,          // de}var matcher = language.NewMatcher(serverLangs)func main() {    tag, index, confidence := matcher.Match(userPrefs...)    fmt.Printf("best match: %s (%s) index=%d confidence=%v\n",        display.English.Tags().Name(tag),        display.Self.Name(tag),        index, confidence)    // best match: German (Deutsch) index=1 confidence=High}

Creating Language Tags

The simplest way to create a language.Tag from a user-given language code string is with language.Make. It extracts meaningful information even from malformed input. For example, “en-USD” will result in “en” even though USD is not a valid subtag.

Make doesn’t return an error. It is common practice to use the default language if an error occurs anyway so this makes it more convenient. Use Parse to handle any error manually.

The HTTP Accept-Language header is often used to pass a user’s desired languages. The ParseAcceptLanguage function parses it into a slice of language tags, ordered by preference.

By default, the language package does not canonicalize tags. For example, it does not follow the BCP 47 recommendation of eliminating scripts if it is the common choice in the “overwhelming majority”. It similarly ignores CLDR recommendations: “cmn” is not replaced by “zh” and“zh-Hant-HK” is not simplified to “zh-HK”. Canonicalizing tags may throw away useful information about user intent. Canonicalization is handled in the Matcher instead. A full array of canonicalization options are available if the programmer still desires to do so.

Matching User-Preferred Languages to Supported Languages

A Matcher matches user-preferred languages to supported languages. Users are strongly advised to use it if they don’t want to deal with all the intricacies of matching languages.

The Match method may pass through user settings (from BCP 47 extensions) from the preferred tags to the selected supported tag. It is therefore important that the tag returned by Match is used to obtain language-specific resources. For example, “de-u-co-phonebk” requests phone-book ordering for German. The extension is ignored for matching, but is used by the collate package to select the respective sorting order variant.

A Matcher is initialized with the languages supported by an application, which are usually the languages for which there are translations. This set is typically fixed, allowing a matcher to be created at startup. Matcher is optimized to improve the performance of Match at the expense of initialization cost.

The language package provides a predefined set of the most commonly used language tags that can be used for defining the supported set. Users generally don’t have to worry about the exact tags to pick for supported languages. For example, AmericanEnglish (“en-US”) may be used interchangeably with the more common English (“en”), which defaults to American. It is all the same for the Matcher. An application may even add both, allowing for more specific American slang for “en-US”.

Matching Example

Consider the following Matcher and lists of supported languages:

var supported = []language.Tag{    language.AmericanEnglish,    // en-US: first language is fallback    language.German,             // de    language.Dutch,              // nl    language.Portuguese          // pt (defaults to Brazilian)    language.EuropeanPortuguese, // pt-pT    language.Romanian            // ro    language.Serbian,            // sr (defaults to Cyrillic script)    language.SerbianLatin,       // sr-Latn    language.SimplifiedChinese,  // zh-Hans    language.TraditionalChinese, // zh-Hant}var matcher = language.NewMatcher(supported)

Let's look at the matches against this list of supported languages for various user preferences.

For a user preference of "he" (Hebrew), the best match is "en-US" (American English). There is no good match, so the matcher uses the fallback language (the first in the supported list).

For a user preference of "hr" (Croatian), the best match is "sr-Latn" (Serbian with Latin script), because, once they are written in the same script, Serbian and Croatian are mutually intelligible.

For a user preference of "ru, mo" (Russian, then Moldavian), the best match is"ro" (Romanian), because Moldavian is now canonically classified as "ro-MD" (Romanian in Moldova).

For a user preference of "zh-TW" (Mandarin in Taiwan), the best match is"zh-Hant" (Mandarin written in Traditional Chinese), not "zh-Hans" (Mandarin written in Simplified Chinese).

For a user preference of "af, ar" (Afrikaans, then Arabic), the best match is"nl" (Dutch). Neither preference is supported directly, but Dutch is a significantly closer match to Afrikaans than the fallback language English is to either.

For a user preference of "pt-AO, id" (Angolan Portuguese, then Indonesian), the best match is "pt-PT" (European Portuguese), not "pt" (Brazilian Portuguese).

For a user preference of "gsw-u-co-phonebk" (Swiss German with phone-book collation order), the best match is "de-u-co-phonebk" (German with phone-book collation order). German is the best match for Swiss German in the server's language list, and the option for phone-book collation order has been carried over.

Confidence Scores

Go uses coarse-grained confidence scoring with rule-based elimination. A match is classified as Exact, High (not exact, but no known ambiguity), Low (probably the correct match, but maybe not), or No. In case of multiple matches, there is a set of tie-breaking rules that are executed in order. The first match is returned in the case of multiple equal matches. These confidence scores may be useful, for example, to reject relatively weak matches. They are also used to score, for example, the most likely region or script from a language tag.

Implementations in other languages often use more fine-grained, variable-scale scoring. We found that using coarse-grained scoring in the Go implementation ended up simpler to implement, more maintainable, and faster, meaning that we could handle more rules.

Displaying Supported Languages

The golang.org/x/text/language/display package allows naming language tags in many languages. It also contains a “Self” namer for displaying a tag in its own language.

For example:

    var supported = []language.Tag{        language.English,            // en        language.French,             // fr        language.Dutch,              // nl        language.Make("nl-BE"),      // nl-BE        language.SimplifiedChinese,  // zh-Hans        language.TraditionalChinese, // zh-Hant        language.Russian,            // ru    }    en := display.English.Tags()    for _, t := range supported {        fmt.Printf("%-20s (%s)\n", en.Name(t), display.Self.Name(t))    }

prints

English              (English)French               (français)Dutch                (Nederlands)Flemish              (Vlaams)Simplified Chinese   (简体中文)Traditional Chinese  (繁體中文)Russian              (русский)

In the second column, note the differences in capitalization, reflecting the rules of the respective language.

Conclusion

At first glance, language tags look like nicely structured data, but because they describe human languages, the structure of relationships between language tags is actually quite complex. It is often tempting, especially for English-speaking programmers, to write ad-hoc language matching using nothing other than string manipulation of the language tags. As described above, this can produce awful results.

Go's golang.org/x/text/language package solves this complex problem while still presenting a simple, easy-to-use API. Enjoy.

Go 1.6 is released

$
0
0

Today we release Go version 1.6, the seventh major stable release of Go. You can grab it right now from the download page. Although the release of Go 1.5 six months ago contained dramatic implementation changes, this release is more incremental.

The most significant change is support for HTTP/2 in the net/http package. HTTP/2 is a new protocol, a follow-on to HTTP that has already seen widespread adoption by browser vendors and major websites. In Go 1.6, support for HTTP/2 is enabled by default for both servers and clients when using HTTPS, bringing the benefits of the new protocol to a wide range of Go projects, such as the popular Caddy web server.

The template packages have learned some new tricks, with support for trimming spaces around template actions to produce cleaner template output, and the introduction of the {{block}} action that can be used to create templates that build on other templates. A new template example program demonstrates these new features.

Go 1.5 introduced experimental support for a “vendor” directory that was enabled by an environment variable. In Go 1.6, the feature is now enabled by default. Source trees that contain a directory named “vendor” that is not used in accordance with the new feature will require changes to avoid broken builds (the simplest fix is to rename the directory).

The runtime has added lightweight, best-effort detection of concurrent misuse of maps. As always, if one goroutine is writing to a map, no other goroutine should be reading or writing the map concurrently. If the runtime detects this condition, it prints a diagnosis and crashes the program. The best way to find out more about the problem is to run it under therace detector, which will more reliably identify the race and give more detail.

The runtime has also changed how it prints program-ending panics. It now prints only the stack of the panicking goroutine, rather than all existing goroutines. This behavior can be configured using theGOTRACEBACK environment variable or by calling the debug.SetTraceback function.

Users of cgo should be aware of major changes to the rules for sharing pointers between Go and C code. The rules are designed to ensure that such C code can coexist with Go's garbage collector and are checked during program execution, so code may require changes to avoid crashes. See the release notes andcgo documentation for the details.

The compiler, linker, and go command have a new -msan flag analogous to -race and only available on linux/amd64, that enables interoperation with theClang MemorySanitizer. This is useful for testing a program containing suspect C or C++ code. You might like to try it while testing your cgo code with the new pointer rules.

Performance of Go programs built with Go 1.6 remains similar to those built with Go 1.5. Garbage-collection pauses are even lower than with Go 1.5, but this is particularly noticeable for programs using large amounts of memory. With regard to the performance of the compiler tool chain, build times should be similar to those of Go 1.5.

The algorithm inside sort.Sort was improved to run about 10% faster, but the change may break programs that expect a specific ordering of equal but distinguishable elements. Such programs should refine their Less methods to indicate the desired ordering or use sort.Stable to preserve the input order for equal values.

And, of course, there are many more additions, improvements, and fixes. You can find them all in the comprehensive release notes.

To celebrate the release,Go User Groups around the world are holding release parties on the 17th of February. Online, the Go contributors are hosting a question and answer session on the golang subreddit for the next 24 hours. If you have questions about the project, the release, or just Go in general, then please join the discussion.

Thanks to everyone that contributed to the release. Happy hacking.


Go 1.7 is released

$
0
0

Today we are happy to announce the release of Go 1.7. You can get it from the download page. There are several significant changes in this release: a port forLinux on IBM z Systems (s390x), compiler improvements, the addition of the context package, and support for hierarchical tests and benchmarks.

A new compiler back end, based on static single-assignment form (SSA), has been under development for the past year. By representing a program in SSA form, a compiler may perform advanced optimizations more easily. This new back end generates more compact, more efficient code that includes optimizations likebounds check elimination andcommon subexpression elimination. We observed a 5–35% speedup across our benchmarks. For now, the new backend is only available for the 64-bit x86 platform ("amd64"), but we’re planning to convert more architecture backends to SSA in future releases.

The compiler front end uses a new, more compact export data format, and processes import declarations more efficiently. While these changes across the compiler toolchain are mostly invisible, users have observed a significant speedup in compile time and a reduction in binary size by as much as 20–30%.

Programs should run a bit faster due to speedups in the garbage collector and optimizations in the standard library. Programs with many idle goroutines will experience much shorter garbage collection pauses than in Go 1.6.

Over the past few years, the golang.org/x/net/context package has proven to be essential to many Go applications. Contexts are used to great effect in applications related to networking, infrastructure, and microservices (such as Kubernetes and Docker). They make it easy to enable cancelation, timeouts, and passing request-scoped data. To make use of contexts within the standard library and to encourage more extensive use, the package has been moved from the x/net repository to the standard library as the context package. Support for contexts has been added to thenet,net/http, andos/exec packages. For more information about contexts, see the package documentation and the Go blog post Go Concurrency Patterns: Context.

Go 1.5 introduced experimental support for a "vendor" directory, enabled by the GO15VENDOREXPERIMENT environment variable. Go 1.6 enabled this behavior by default, and in Go 1.7, this switch has been removed and the "vendor" behavior is always enabled.

Go 1.7 includes many more additions, improvements, and fixes. Find the complete set of changes, and details of the points above, in theGo 1.7 release notes.

Finally, the Go team would like thank everyone who contributed to the release. 170 people contributed to this release, including 140 from the Go community. These contributions ranged from changes to the compiler and linker, to the standard library, to documentation, and code reviews. We welcome contributions; if you'd like to get involved, check out thecontribution guidelines.

Smaller Go 1.7 binaries

$
0
0

Introduction

Go was designed for writing servers. That is how it is most widely used today, and as a result a lot of work on the runtime and compiler is focused on issues that matter to servers: latency, ease of deployment, precise garbage collection, fast startup time, performance.

As Go gets used for a wider variety of programs, there are new issues that must be considered. One of these is binary size. It has been on the radar for a long time (issue #6853 was filed over two years ago), but the growing interest in using Go for deploying binaries on smaller devices — such as the Raspberry Pi or mobile devices — means it received some attention for the Go 1.7 release.

Work done in Go 1.7

Three significant changes in Go 1.7 affect binary size.

The first is the new SSA backend that was enabled for AMD64 in this release. While the primary motivation for SSA was improved performance, the better generated code is also smaller. The SSA backend shrinks Go binaries by ~5%. We expect larger gains for the more RISC-like architectures like ARM and MIPS when those backends have been converted to SSA in Go 1.8.

The second change is method pruning. Until 1.6, all methods on all used types were kept, even if some of the methods were never called. This is because they might be called through an interface, or called dynamically using the reflect package. Now the compiler discards any unexported methods that do not match an interface. Similarly the linker can discard other exported methods, those that are only accessible through reflection, if the correspondingreflection features are not used anywhere in the program. That change shrinks binaries by 5–20%.

The third change is a more compact format for run-time type information used by the reflect package. The encoding format was originally designed to make the decoder in the runtime and reflect packages as simple as possible. By making this code a bit harder to read we can compress the format without affecting the run-time performance of Go programs. The new format shrinks Go binaries by a further 5–15%. Libraries built for Android and archives built for iOS shrink further as the new format contains fewer pointers, each of which requires dynamic relocations in position independent code.

In addition, there were many small improvements such as improved interface data layout, better static data layout, and simplified dependencies. For example, the HTTP client no longer links in the entire HTTP server. The full list of changes can be found in issue#6853.

Results

Typical programs, ranging from tiny toys to large production programs, are about 30% smaller when built with Go 1.7.

The canonical hello world program goes from 2.3MB to 1.6MB:

package mainimport "fmt"func main() {    fmt.Println("Hello, World!")}

When compiled without debugging information the statically linked binary is now under a megabyte.

A large production program used for testing this cycle, jujud, went from 94MB to 67MB.

Position-independent binaries are 50% smaller.

In a position-independent executable (PIE), a pointer in a read-only data section requires a dynamic relocation. Because the new format for type information replaces pointers by section offsets, it saves 28 bytes per pointer.

Position-independent executables with debugging information removed are particularly important to mobile developers, as this is the kind of program shipped to phones. Big downloads make for a poor user experience, so the reduction here is good news.

Future Work

Several changes to the run-time type information were too late for the Go 1.7 freeze, but will hopefully make it into 1.8, further shrinking programs, especially position-independent ones.

These changes are all conservative, reducing binary size without increasing build time, startup time, overall execution time, or memory usage. We could take more radical steps to reduce binary size: theupx tool for compressing executables shrinks binaries by another 50% at the cost of increased startup time and potentially increased memory use. For extremely small systems (the kind that might live on a keychain) we could build a version of Go without reflection, though it is unclear whether such a restricted language would be sufficiently useful. For some algorithms in the runtime we could use slower but more compact implementions when every kilobyte counts. All of these call for more research in later development cycles.

To the many contributors who helped make Go 1.7 binaries smaller, thank you!

Using Subtests and Sub-benchmarks

$
0
0

Introduction

In Go 1.7, the testing package introduces a Run method on theT andB types that allows for the creation of subtests and sub-benchmarks. The introduction of subtests and sub-benchmarks enables better handling of failures, fine-grained control of which tests to run from the command line, control of parallelism, and often results in simpler and more maintainable code.

Table-driven tests basics

Before digging into the details, let's first discuss a common way of writing tests in Go. A series of related checks can be implemented by looping over a slice of test cases:

func TestTime(t *testing.T) {    testCases := []struct {        gmt  string        loc  string        want string    }{        {"12:31", "Europe/Zuri", "13:31"},     // incorrect location name        {"12:31", "America/New_York", "7:31"}, // should be 07:31        {"08:08", "Australia/Sydney", "18:08"},    }    for _, tc := range testCases {        loc, err := time.LoadLocation(tc.loc)        if err != nil {            t.Fatalf("could not load location %q", tc.loc)        }        gmt, _ := time.Parse("15:04", tc.gmt)        if got := gmt.In(loc).Format("15:04"); got != tc.want {            t.Errorf("In(%s, %s) = %s; want %s", tc.gmt, tc.loc, got, tc.want)        }    }}

This approach, commonly referred to as table-driven tests, reduces the amount of repetitive code compared to repeating the same code for each test and makes it straightforward to add more test cases.

Table-driven benchmarks

Before Go 1.7 it was not possible to use the same table-driven approach for benchmarks. A benchmark tests the performance of an entire function, so iterating over benchmarks would just measure all of them as a single benchmark.

A common workaround was to define separate top-level benchmarks that each call a common function with different parameters. For instance, before 1.7 the strconv package's benchmarks for AppendFloat looked something like this:

func benchmarkAppendFloat(b *testing.B, f float64, fmt byte, prec, bitSize int) {    dst := make([]byte, 30)    b.ResetTimer() // Overkill here, but for illustrative purposes.    for i := 0; i < b.N; i++ {        AppendFloat(dst[:0], f, fmt, prec, bitSize)    }}func BenchmarkAppendFloatDecimal(b *testing.B) { benchmarkAppendFloat(b, 33909, 'g', -1, 64) }func BenchmarkAppendFloat(b *testing.B)        { benchmarkAppendFloat(b, 339.7784, 'g', -1, 64) }func BenchmarkAppendFloatExp(b *testing.B)     { benchmarkAppendFloat(b, -5.09e75, 'g', -1, 64) }func BenchmarkAppendFloatNegExp(b *testing.B)  { benchmarkAppendFloat(b, -5.11e-95, 'g', -1, 64) }func BenchmarkAppendFloatBig(b *testing.B)     { benchmarkAppendFloat(b, 123456789123456789123456789, 'g', -1, 64) }...

Using the Run method available in Go 1.7, the same set of benchmarks is now expressed as a single top-level benchmark:

func BenchmarkAppendFloat(b *testing.B) {    benchmarks := []struct{        name    string        float   float64        fmt     byte        prec    int        bitSize int    }{        {"Decimal", 33909, 'g', -1, 64},        {"Float", 339.7784, 'g', -1, 64},        {"Exp", -5.09e75, 'g', -1, 64},        {"NegExp", -5.11e-95, 'g', -1, 64},        {"Big", 123456789123456789123456789, 'g', -1, 64},        ...    }    dst := make([]byte, 30)    for _, bm := range benchmarks {        b.Run(bm.name, func(b *testing.B) {            for i := 0; i < b.N; i++ {                AppendFloat(dst[:0], bm.float, bm.fmt, bm.prec, bm.bitSize)            }        })    }}

Each invocation of the Run method creates a separate benchmark. An enclosing benchmark function that calls a Run method is only run once and is not measured.

The new code has more lines of code, but is more maintainable, more readable, and consistent with the table-driven approach commonly used for testing. Moreover, common setup code is now shared between runs while eliminating the need to reset the timer.

Table-driven tests using subtests

Go 1.7 also introduces a Run method for creating subtests. This test is a rewritten version of our earlier example using subtests:

func TestTime(t *testing.T) {    testCases := []struct {        gmt  string        loc  string        want string    }{        {"12:31", "Europe/Zuri", "13:31"},        {"12:31", "America/New_York", "7:31"},        {"08:08", "Australia/Sydney", "18:08"},    }    for _, tc := range testCases {        t.Run(fmt.Sprintf("%s in %s", tc.gmt, tc.loc), func(t *testing.T) {            loc, err := time.LoadLocation(tc.loc)            if err != nil {                t.Fatal("could not load location")            }            gmt, _ := time.Parse("15:04", tc.gmt)            if got := gmt.In(loc).Format("15:04"); got != tc.want {                t.Errorf("got %s; want %s", got, tc.want)            }        })    }}

The first thing to note is the difference in output from the two implementations. The original implementation prints:

--- FAIL: TestTime (0.00s)    time_test.go:62: could not load location "Europe/Zuri"

Even though there are two errors, execution of the test halts on the call toFatalf and the second test never runs.

The implementation using Run prints both:

--- FAIL: TestTime (0.00s)    --- FAIL: TestTime/12:31_in_Europe/Zuri (0.00s)        time_test.go:84: could not load location    --- FAIL: TestTime/12:31_in_America/New_York (0.00s)        time_test.go:88: got 07:31; want 7:31

Fatal and its siblings causes a subtest to be skipped but not its parent or subsequent subtests.

Another thing to note is the shorter error messages in the new implementation. Since the subtest name uniquely identifies the subtest there is no need to identify the test again within the error messages.

There are several other benefits to using subtests or sub-benchmarks, as clarified by the following sections.

Running specific tests or benchmarks

Both subtests and sub-benchmarks can be singled out on the command line using the -run or -bench flag. Both flags take a slash-separated list of regular expressions that match the corresponding parts of the full name of the subtest or sub-benchmark.

The full name of a subtest or sub-benchmark is a slash-separated list of its name and the names of all of its parents, starting with the top-level. The name is the corresponding function name for top-level tests and benchmarks, and the first argument to Run otherwise. To avoid display and parsing issues, a name is sanitized by replacing spaces with underscores and escaping non-printable characters. The same sanitizing is applied to the regular expressions passed to the -run or -bench flags.

A few examples:

Run tests that use a timezone in Europe:

$ go test -run=TestTime/"in Europe"--- FAIL: TestTime (0.00s)    --- FAIL: TestTime/12:31_in_Europe/Zuri (0.00s)        time_test.go:85: could not load location

Run only tests for times after noon:

$ go test -run=Time/12:[0-9] -v=== RUN   TestTime=== RUN   TestTime/12:31_in_Europe/Zuri=== RUN   TestTime/12:31_in_America/New_York--- FAIL: TestTime (0.00s)    --- FAIL: TestTime/12:31_in_Europe/Zuri (0.00s)        time_test.go:85: could not load location    --- FAIL: TestTime/12:31_in_America/New_York (0.00s)        time_test.go:89: got 07:31; want 7:31

Perhaps a bit surprising, using -run=TestTime/New_York won't match any tests. This is because the slash present in the location names is treated as a separator as well. Instead use:

$ go test -run=Time//New_York--- FAIL: TestTime (0.00s)    --- FAIL: TestTime/12:31_in_America/New_York (0.00s)        time_test.go:88: got 07:31; want 7:31

Note the // in the string passed to -run. The / in time zone name America/New_York is handled as if it were a separator resulting from a subtest. The first regular expression of the pattern (TestTime) matches the top-level test. The second regular expression (the empty string) matches anything, in this case the time and the continent part of the location. The third regular expression (New_York) matches the city part of the location.

Treating slashes in names as separators allows the user to refactor hierarchies of tests without the need to change the naming. It also simplifies the escaping rules. The user should escape slashes in names, for instance by replacing them with backslashes, if this poses a problem.

A unique sequence number is appended to test names that are not unique. So one could just pass an empty string to Run if there is no obvious naming scheme for subtests and the subtests can easily be identified by their sequence number.

Setup and Tear-down

Subtests and sub-benchmarks can be used to manage common setup and tear-down code:

func TestFoo(t *testing.T) {    // <setup code>    t.Run("A=1", func(t *testing.T) { ... })    t.Run("A=2", func(t *testing.T) { ... })    t.Run("B=1", func(t *testing.T) {        if !test(foo{B:1}) {            t.Fail()        }    })    // <tear-down code>}

The setup and tear-down code will run if any of the enclosed subtests are run and will run at most once. This applies even if any of the subtests calls Skip, Fail, or Fatal.

Control of Parallelism

Subtests allow fine-grained control over parallelism. To understand how to use subtests in the way it is important to understand the semantics of parallel tests.

Each test is associated with a test function. A test is called a parallel test if its test function calls the Parallel method on its instance of testing.T. A parallel test never runs concurrently with a sequential test and its execution is suspended until its calling test function, that of the parent test, has returned. The -parallel flag defines the maximum number of parallel tests that can run in parallel.

A test blocks until its test function returns and all of its subtests have completed. This means that the parallel tests that are run by a sequential test will complete before any other consecutive sequential test is run.

This behavior is identical for tests created by Run and top-level tests. In fact, under the hood top-level tests are implemented as subtests of a hidden master test.

Run a group of tests in parallel

The above semantics allows for running a group of tests in parallel with each other but not with other parallel tests:

func TestGroupedParallel(t *testing.T) {    for _, tc := range testCases {        tc := tc // capture range variable        t.Run(tc.Name, func(t *testing.T) {            t.Parallel()            if got := foo(tc.in); got != tc.out {                t.Errorf("got %v; want %v", got, tc.out)            }            ...        })    }}

The outer test will not complete until all parallel tests started by Run have completed. As a result, no other parallel tests can run in parallel to these parallel tests.

Note that we need to capture the range variable to ensure that tc gets bound to the correct instance.

Cleaning up after a group of parallel tests

In the previous example we used the semantics to wait on a group of parallel tests to complete before commencing other tests. The same technique can be used to clean up after a group of parallel tests that share common resources:

func TestTeardownParallel(t *testing.T) {    // <setup code>    // This Run will not return until its parallel subtests complete.    t.Run("group", func(t *testing.T) {        t.Run("Test1", parallelTest1)        t.Run("Test2", parallelTest2)        t.Run("Test3", parallelTest3)    })    // <tear-down code>}

The behavior of waiting on a group of parallel tests is identical to that of the previous example.

Conclusion

Go 1.7's addition of subtests and sub-benchmarks allows you to write structured tests and benchmarks in a natural way that blends nicely into the existing tools. One way to think about this is that earlier versions of the testing package had a 1-level hierarchy: the package-level test was structured as a set of individual tests and benchmarks. Now that structure has been extended to those individual tests and benchmarks, recursively. In fact, in the implementation, the top-level tests and benchmarks are tracked as if they were subtests and sub-benchmarks of an implicit master test and benchmark: the treatment really is the same at all levels.

The ability for tests to define this structure enables fine-grained execution of specific test cases, shared setup and teardown, and better control over test parallelism. We are excited to see what other uses people find. Enjoy.

Introducing HTTP Tracing

$
0
0

Introduction

In Go 1.7 we introduced HTTP tracing, a facility to gather fine-grained information throughout the lifecycle of an HTTP client request. Support for HTTP tracing is provided by the net/http/httptrace package. The collected information can be used for debugging latency issues, service monitoring, writing adaptive systems, and more.

HTTP events

The httptrace package provides a number of hooks to gather information during an HTTP round trip about a variety of events. These events include:

  • Connection creation
  • Connection reuse
  • DNS lookups
  • Writing the request to the wire
  • Reading the response

Tracing events

You can enable HTTP tracing by putting an*httptrace.ClientTrace containing hook functions into a request's context.Context. Various http.RoundTripper implementations report the internal events by looking for context's *httptrace.ClientTrace and calling the relevant hook functions.

The tracing is scoped to the request's context and users should put a *httptrace.ClientTrace to the request context before they start a request.

    req, _ := http.NewRequest("GET", "http://example.com", nil)    trace := &httptrace.ClientTrace{        DNSDone: func(dnsInfo httptrace.DNSDoneInfo) {            fmt.Printf("DNS Info: %+v\n", dnsInfo)        },        GotConn: func(connInfo httptrace.GotConnInfo) {            fmt.Printf("Got Conn: %+v\n", connInfo)        },    }    req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))    if _, err := http.DefaultTransport.RoundTrip(req); err != nil {        log.Fatal(err)    }

During a round trip, http.DefaultTransport will invoke each hook as an event happens. The program above will print the DNS information as soon as the DNS lookup is complete. It will similarly print connection information when a connection is established to the request's host.

Tracing with http.Client

The tracing mechanism is designed to trace the events in the lifecycle of a single http.Transport.RoundTrip. However, a client may make multiple round trips to complete an HTTP request. For example, in the case of a URL redirection, the registered hooks will be called as many times as the client follows HTTP redirects, making multiple requests. Users are responsible for recognizing such events at the http.Client level. The program below identifies the current request by using anhttp.RoundTripper wrapper.

package mainimport ("fmt""log""net/http""net/http/httptrace")// transport is an http.RoundTripper that keeps track of the in-flight// request and implements hooks to report HTTP tracing events.type transport struct {    current *http.Request}// RoundTrip wraps http.DefaultTransport.RoundTrip to keep track// of the current request.func (t *transport) RoundTrip(req *http.Request) (*http.Response, error) {    t.current = req    return http.DefaultTransport.RoundTrip(req)}// GotConn prints whether the connection has been used previously// for the current request.func (t *transport) GotConn(info httptrace.GotConnInfo) {    fmt.Printf("Connection reused for %v? %v\n", t.current.URL, info.Reused)}func main() {    t := &transport{}    req, _ := http.NewRequest("GET", "https://google.com", nil)    trace := &httptrace.ClientTrace{        GotConn: t.GotConn,    }    req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))    client := &http.Client{Transport: t}    if _, err := client.Do(req); err != nil {        log.Fatal(err)    }}

The program will follow the redirect of google.com to www.google.com and will output:

Connection reused for https://google.com? falseConnection reused for https://www.google.com/? false

The Transport in the net/http package supports tracing of both HTTP/1 and HTTP/2 requests.

If you are an author of a custom http.RoundTripper implementation, you can support tracing by checking the request context for an*httptest.ClientTrace and invoking the relevant hooks as the events occur.

Conclusion

HTTP tracing is a valuable addition to Go for those who are interested in debugging HTTP request latency and writing tools for network debugging for outbound traffic. By enabling this new facility, we hope to see HTTP debugging, benchmarking and visualization tools from the community — such ashttpstat.

Seven years of Go

$
0
0

Today marks seven years since we open-sourced our preliminary sketch of Go. With the help of the open source community, including more than a thousand individual contributors to the Go source repositories, Go has matured into a language used all over the world.

The most significant user-facing changes to Go over the past year are the addition of built-in support forHTTP/2 inGo 1.6 and the integration of thecontext package into the standard library in Go 1.7. But we’ve been making many less visible improvements. Go 1.7 changed the x86-64 compiler to use a new SSA-based back end, improving the performance of most Go programs by 10–20%. For Go 1.8, planned for release next February, we have changed the compilers for the other architectures to use the new back end too. We’ve also added new ports, to Android on 32-bit x86, Linux on 64-bit MIPS, and Linux on IBM z Systems. And we’ve developed new garbage-collection techniques that reduce typical“stop the world” pauses to under 100 microseconds. (Contrast that with Go 1.5’s big news of 10 milliseconds or less.)

This year kicked off with a global Go hackathon, the Gopher Gala, in January. Then there were Go conferences in India and Dubai in February, China and Japan in April, San Francisco in May, Denver in July, London in August, Paris last month, and Brazil this past weekend. And GothamGo in New York is next week. This year also saw more than 30 new Go user groups, eight new Women Who Go chapters, and four GoBridge workshops around the world.

We continue to be overwhelmed by and grateful for the enthusiasm and support of the Go community. Whether you participate by contributing changes, reporting bugs, sharing your expertise in design discussions, writing blog posts or books, running meetups, helping others learn or improve, open sourcing Go packages you wrote, or just being part of the Go community, the Go team thanks you for your help, your time, and your energy. Go would not be the success it is today without you.

Thank you, and here’s to another year of fun and success with Go!

Viewing all 267 articles
Browse latest View live