Introduction

In 2019, we launched go.dev, a new hub for Go developers.

As part of the site, we also launched pkg.go.dev, a central source of information about Go packages and modules. Likegodoc.org, pkg.go.dev serves Go documentation. However, it also understands modules and has information about past versions of a package!

Throughout this year, we will be adding features topkg.go.dev to help our users better understand their dependencies and help them make better decisions around what libraries to import.

Redirecting godoc.org requests to pkg.go.dev

To minimize confusion about which site to use, later this year we are planning to redirect traffic from godoc.org to the corresponding page on pkg.go.dev. We need your help to ensure that pkg.go.dev addresses all of our users' needs. We encourage everyone to begin using pkg.go.dev today for all of their needs and provide feedback.

Your feedback will inform our transition plan, with the goal of makingpkg.go.dev our primary source of information and documentation for packages and modules. We’re sure there are things that you want to see on pkg.go.dev, and we want to hear from you about what those features are!

You can share your feedback with us on these channels:

• Post on the Go issue tracker.
• Email go-discovery-feedback@google.com.
• Click “Share Feedback” or “Report an Issue” in the go.dev footer.

As part of this transition, we will also be discussing plans for API access topkg.go.dev. We will be posting updates onGo issue 33654.

Frequently asked questions

Since our launch in November, we’ve received tons of great feedback aboutpkg.go.dev from Go users. For the remainder of this post, we thought it would be helpful to answer some frequently asked questions.

My package doesn’t show up on pkg.go.dev! How do I add it?

We monitor the Go Module Index regularly for new packages to add to pkg.go.dev. If you don’t see a package on pkg.go.dev, you can add it by fetching the module version fromproxy.golang.org. Seego.dev/about for instructions.

My package has license restrictions. What’s wrong with it?

We understand it can be a frustrating experience to not be able to see the package you want in its entirety on pkg.go.dev. We appreciate your patience as we improve our license detection algorithm.

Since our launch in November, we've made the following improvements:

• Updated our license policy to include the list of licenses that we detect and recognize
• Worked with the licensecheck team to improve detection for copyright notices
• Established a manual review process for special cases

As always, our license policy is atpkg.go.dev/license-policy. If you are having issues, feel free to file an issue on the Go issue tracker, or emailgo-discovery-feedback@google.com so that we can work with you directly!

Will pkg.go.dev be open-sourced so I can run it at work for my private code?

We understand that corporations with private code want to run a documentation server that provides module support. We want to help meet that need, but we feel we don’t yet understand it as well as we need to.

We’ve heard from users that running the godoc.org server is more complex than it should be, because it is designed for serving at public internet scale instead of just within a company. We believe the currentpkg.go.dev server would have the same problem.

We think a new server is more likely to be the right answer for use with private code, instead of exposing every company to the complexity of running the internet-scale pkg.go.dev codebase. In addition to serving documentation, a new server could also serve information togoimports andgopls.

If you want to run such a server, please fill out this3-5 minute survey to help us better understand your needs. This survey will be available until March 1st, 2020.

We’re excited about the future of pkg.go.dev in 2020, and we hope you are too! We look forward to hearing your feedback and working with the Go community on this transition.

Today the Go team is very happy to announce the release of Go 1.14. You can get it from the download page.

Some of the highlights include:

• Module support in the go command is now ready for production use. We encourage all users to migrate to go modules for dependency management.
• Embedding interfaces with overlapping method sets
• Improved defer performance
• Goroutines are asynchronously preemptible
• The page allocator is more efficient
• Internal timers are more efficient

For the complete list of changes and more information about the improvements above, see the Go 1.14 release notes.

We want to thank everyone who contributed to this release by writing code, filing bugs, providing feedback, and/or testing the beta and release candidate. Your contributions and diligence helped to ensure that Go 1.14 is as stable as possible. That said, if you notice any problems, please file an issue.

We hope you enjoy the new release!

Introduction

We are pleased to announce the release of a major revision of the Go API forprotocol buffers, Google's language-neutral data interchange format.

Motivations for a new API

The first protocol buffer bindings for Go wereannounced by Rob Pike in March of 2010. Go 1 would not be released for another two years.

In the decade since that first release, the package has grown and developed along with Go. Its users' requirements have grown too.

Many people want to write programs that use reflection to examine protocol buffer messages. Thereflect package provides a view of Go types and values, but omits information from the protocol buffer type system. For example, we might want to write a function that traverses a log entry and clears any field annotated as containing sensitive data. The annotations are not part of the Go type system.

Another common desire is to use data structures other than the ones generated by the protocol buffer compiler, such as a dynamic message type capable of representing messages whose type is not known at compile time.

We also observed that a frequent source of problems was that theproto.Message interface, which identifies values of generated message types, does very little to describe the behavior of those types. When users create types that implement that interface (often inadvertently by embedding a message in another struct) and pass values of those types to functions expecting a generated message value, programs crash or behave unpredictably.

All three of these problems have a common cause, and a common solution: The Message interface should fully specify the behavior of a message, and functions operating on Message values should freely accept any type that correctly implements the interface.

Since it is not possible to change the existing definition of theMessage type while keeping the package API compatible, we decided that it was time to begin work on a new, incompatible major version of the protobuf module.

Today, we're pleased to release that new module. We hope you like it.

Reflection

Reflection is the flagship feature of the new implementation. Similar to how the reflect package provides a view of Go types and values, thegoogle.golang.org/protobuf/reflect/protoreflect package provides a view of values according to the protocol buffer type system.

A complete description of the protoreflect package would run too long for this post, but let's look at how we might write the log-scrubbing function we mentioned previously.

First, we'll write a .proto file defining an extension of thegoogle.protobuf.FieldOptions type so we can annotate fields as containing sensitive information or not.

syntax = "proto3";import "google/protobuf/descriptor.proto";package golang.example.policy;extend google.protobuf.FieldOptions {    bool non_sensitive = 50000;}

We can use this option to mark certain fields as non-sensitive.

message MyMessage {    string public_name = 1 [(golang.example.policy.non_sensitive) = true];}

Next, we will write a Go function which accepts an arbitrary message value and removes all the sensitive fields.

// Redact clears every sensitive field in pb.func Redact(pb proto.Message) {   // ...}

This function accepts aproto.Message, an interface type implemented by all generated message types. This type is an alias for one defined in the protoreflect package:

type ProtoMessage interface{    ProtoReflect() Message}

To avoid filling up the namespace of generated messages, the interface contains only a single method returning aprotoreflect.Message, which provides access to the message contents.

(Why an alias? Because protoreflect.Message has a corresponding method returning the original proto.Message, and we need to avoid an import cycle between the two packages.)

Theprotoreflect.Message.Range method calls a function for every populated field in a message.

m := pb.ProtoReflect()m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {    // ...    return true})

The range function is called with aprotoreflect.FieldDescriptor describing the protocol buffer type of the field, and aprotoreflect.Value containing the field value.

Theprotoreflect.FieldDescriptor.Options method returns the field options as a google.protobuf.FieldOptions message.

opts := fd.Options().(*descriptorpb.FieldOptions)

(Why the type assertion? Since the generated descriptorpb package depends on protoreflect, the protoreflect package can't return the concrete options type without causing an import cycle.)

We can then check the options to see the value of our extension boolean:

if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {    return true // don't redact non-sensitive fields}

Note that we are looking at the field descriptor here, not the fieldvalue. The information we're interested in lies in the protocol buffer type system, not the Go one.

This is also an example of an area where we have simplified the proto package API. The originalproto.GetExtension returned both a value and an error. The newproto.GetExtension returns just a value, returning the default value for the field if it is not present. Extension decoding errors are reported at Unmarshal time.

Once we have identified a field that needs redaction, clearing it is simple:

m.Clear(fd)

Putting all the above together, our complete redaction function is:

// Redact clears every sensitive field in pb.func Redact(pb proto.Message) {    m := pb.ProtoReflect()    m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {        opts := fd.Options().(*descriptorpb.FieldOptions)        if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {            return true        }        m.Clear(fd)        return true    })}

A more complete implementation might recursively descend into message-valued fields. We hope that this simple example gives a taste of protocol buffer reflection and its uses.

Versions

We call the original version of Go protocol buffers APIv1, and the new one APIv2. Because APIv2 is not backwards compatible with APIv1, we need to use different module paths for each.

(These API versions are not the same as the versions of the protocol buffer language: proto1, proto2, and proto3. APIv1 and APIv2 are concrete implementations in Go that both support the proto2 andproto3 language versions.)

Thegithub.com/golang/protobuf module is APIv1.

Thegoogle.golang.org/protobuf module is APIv2. We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider. (We considered google.golang.org/protobuf/v2, to make it clear that this is the second major version of the API, but settled on the shorter path as being the better choice in the long term.)

We know that not all users will move to a new major version of a package at the same rate. Some will switch quickly; others may remain on the old version indefinitely. Even within a single program, some parts may use one API while others use another. It is essential, therefore, that we continue to support programs that use APIv1.

• github.com/golang/protobuf@v1.3.4 is the most recent pre-APIv2 version of APIv1.

• github.com/golang/protobuf@v1.4.0 is a version of APIv1 implemented in terms of APIv2. The API is the same, but the underlying implementation is backed by the new one. This version contains functions to convert between the APIv1 and APIv2 proto.Message interfaces to ease the transition between the two.

• google.golang.org/protobuf@v1.20.0 is APIv2. This module depends upon github.com/golang/protobuf@v1.4.0, so any program which uses APIv2 will automatically pick a version of APIv1 which integrates with it.

(Why start at version v1.20.0? To provide clarity. We do not anticipate APIv1 to ever reach v1.20.0, so the version number alone should be enough to unambiguously differentiate between APIv1 and APIv2.)

We intend to maintain support for APIv1 indefinitely.

This organization ensures that any given program will use only a single protocol buffer implementation, regardless of which API version it uses. It permits programs to adopt the new API gradually, or not at all, while still gaining the advantages of the new implementation. The principle of minimum version selection means that programs may remain on the old implementation until the maintainers choose to update to the new one (either directly, or by updating a dependency).

Additional features of note

Thegoogle.golang.org/protobuf/encoding/protojson package converts protocol buffer messages to and from JSON using thecanonical JSON mapping, and fixes a number of issues with the old jsonpb package that were difficult to change without causing problems for existing users.

Thegoogle.golang.org/protobuf/types/dynamicpb package provides an implementation of proto.Message for messages whose protocol buffer type is derived at runtime.

Thegoogle.golang.org/protobuf/testing/protocmp package provides functions to compare protocol buffer messages with thegithub.com/google/cmp package.

Thegoogle.golang.org/protobuf/compiler/protogen package provides support for writing protocol compiler plugins.

Conclusion

The google.golang.org/protobuf module is a major overhaul of Go's support for protocol buffers, providing first-class support for reflection, custom message implementations, and a cleaned up API surface. We intend to maintain the previous API indefinitely as a wrapper of the new one, allowing users to adopt the new API incrementally at their own pace.

Our goal in this update is to improve upon the benefits of the old API while addressing its shortcomings. As we completed each component of the new implementation, we put it into use within Google's codebase. This incremental rollout has given us confidence in both the usability of the new API and the performance and correctness of the new implementation. We believe it is production ready.

We are excited about this release and hope that it will serve the Go ecosystem for the next ten years and beyond!

Introduction

We are pleased to announce the release of a major revision of the Go API forprotocol buffers,Google's language-neutral data interchange format.

Motivations for a new API

The first protocol buffer bindings for Go wereannounced by Rob Pikein March of 2010. Go 1 would not be released for another two years.

In the decade since that first release, the package has grown anddeveloped along with Go. Its users' requirements have grown too.

Many people want to write programs that use reflection to examine protocolbuffer messages. Thereflectpackage provides a view of Go types andvalues, but omits information from the protocol buffer type system. Forexample, we might want to write a function that traverses a log entry andclears any field annotated as containing sensitive data. The annotationsare not part of the Go type system.

Another common desire is to use data structures other than the onesgenerated by the protocol buffer compiler, such as a dynamic message typecapable of representing messages whose type is not known at compile time.

We also observed that a frequent source of problems was that theproto.Messageinterface, which identifies values of generated message types, does verylittle to describe the behavior of those types. When users create typesthat implement that interface (often inadvertently by embedding a messagein another struct) and pass values of those types to functions expectinga generated message value, programs crash or behave unpredictably.

All three of these problems have a common cause, and a common solution:The Message interface should fully specify the behavior of a message,and functions operating on Message values should freely accept anytype that correctly implements the interface.

Since it is not possible to change the existing definition of theMessage type while keeping the package API compatible, we decided thatit was time to begin work on a new, incompatible major version of theprotobuf module.

Today, we're pleased to release that new module. We hope you like it.

Reflection

Reflection is the flagship feature of the new implementation. Similarto how the reflect package provides a view of Go types and values, thegoogle.golang.org/protobuf/reflect/protoreflectpackage provides a view of values according to the protocol buffertype system.

A complete description of the protoreflect package would run too longfor this post, but let's look at how we might write the log-scrubbingfunction we mentioned previously.

First, we'll write a .proto file defining an extension of thegoogle.protobuf.FieldOptionstype so we can annotate fields as containingsensitive information or not.

syntax = "proto3";import "google/protobuf/descriptor.proto";package golang.example.policy;extend google.protobuf.FieldOptions {    bool non_sensitive = 50000;}

We can use this option to mark certain fields as non-sensitive.

message MyMessage {    string public_name = 1 [(golang.example.policy.non_sensitive) = true];}

Next, we will write a Go function which accepts an arbitrary messagevalue and removes all the sensitive fields.

// Redact clears every sensitive field in pb.func Redact(pb proto.Message) {   // ...}

This function accepts aproto.Message,an interface type implemented by all generated message types. This typeis an alias for one defined in the protoreflect package:

type ProtoMessage interface{    ProtoReflect() Message}

To avoid filling up the namespace of generatedmessages, the interface contains only a single method returning aprotoreflect.Message,which provides access to the message contents.

(Why an alias? Because protoreflect.Message has a correspondingmethod returning the original proto.Message, and we need to avoid animport cycle between the two packages.)

Theprotoreflect.Message.Rangemethod calls a function for every populated field in a message.

m := pb.ProtoReflect()m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {    // ...    return true})

The range function is called with aprotoreflect.FieldDescriptordescribing the protocol buffer type of the field, and aprotoreflect.Valuecontaining the field value.

Theprotoreflect.FieldDescriptor.Optionsmethod returns the field options as a google.protobuf.FieldOptionsmessage.

opts := fd.Options().(*descriptorpb.FieldOptions)

(Why the type assertion? Since the generated descriptorpb packagedepends on protoreflect, the protoreflect package can't return theconcrete options type without causing an import cycle.)

We can then check the options to see the value of our extension boolean:

if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {    return true // don't redact non-sensitive fields}

Note that we are looking at the field descriptor here, not the fieldvalue. The information we're interested in lies in the protocolbuffer type system, not the Go one.

This is also an example of an area where wehave simplified the proto package API. The originalproto.GetExtensionreturned both a value and an error. The newproto.GetExtensionreturns just a value, returning the default value for the field if it isnot present. Extension decoding errors are reported at Unmarshal time.

Once we have identified a field that needs redaction, clearing it is simple:

m.Clear(fd)

Putting all the above together, our complete redaction function is:

// Redact clears every sensitive field in pb.func Redact(pb proto.Message) {    m := pb.ProtoReflect()    m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {        opts := fd.Options().(*descriptorpb.FieldOptions)        if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {            return true        }        m.Clear(fd)        return true    })}

A more complete implementation might recursively descend intomessage-valued fields. We hope that this simple example gives ataste of protocol buffer reflection and its uses.

Versions

We call the original version of Go protocol buffers APIv1, and thenew one APIv2. Because APIv2 is not backwards compatible with APIv1,we need to use different module paths for each.

(These API versions are not the same as the versions of the protocolbuffer language: proto1, proto2, and proto3. APIv1 and APIv2are concrete implementations in Go that both support the proto2 andproto3 language versions.)

Thegithub.com/golang/protobufmodule is APIv1.

Thegoogle.golang.org/protobufmodule is APIv2. We have taken advantage of the need to change theimport path to switch to one that is not tied to a specific hostingprovider. (We considered google.golang.org/protobuf/v2, to make itclear that this is the second major version of the API, but settled onthe shorter path as being the better choice in the long term.)

We know that not all users will move to a new major version of a packageat the same rate. Some will switch quickly; others may remain on the oldversion indefinitely. Even within a single program, some parts may useone API while others use another. It is essential, therefore, that wecontinue to support programs that use APIv1.

• github.com/golang/protobuf@v1.3.4 is the most recent pre-APIv2 version of APIv1.

• github.com/golang/protobuf@v1.4.0 is a version of APIv1 implemented in terms of APIv2.The API is the same, but the underlying implementation is backed by the new one.This version contains functions to convert between the APIv1 and APIv2 proto.Messageinterfaces to ease the transition between the two.

• google.golang.org/protobuf@v1.20.0 is APIv2.This module depends upon github.com/golang/protobuf@v1.4.0,so any program which uses APIv2 will automatically pick a version of APIv1which integrates with it.

(Why start at version v1.20.0? To provide clarity.We do not anticipate APIv1 to ever reach v1.20.0,so the version number alone should be enough to unambiguously differentiatebetween APIv1 and APIv2.)

We intend to maintain support for APIv1 indefinitely.

This organization ensures that any given program will use only a singleprotocol buffer implementation, regardless of which API version it uses.It permits programs to adopt the new API gradually, or not at all, whilestill gaining the advantages of the new implementation. The principle ofminimum version selection means that programs may remain on the oldimplementation until the maintainers choose to update to the new one(either directly, or by updating a dependency).

Additional features of note

Thegoogle.golang.org/protobuf/encoding/protojsonpackage converts protocol buffer messages to and from JSON using thecanonical JSON mapping,and fixes a number of issues with the old jsonpb packagethat were difficult to change without causing problems for existing users.

Thegoogle.golang.org/protobuf/types/dynamicpbpackage provides an implementation of proto.Message for messages whoseprotocol buffer type is derived at runtime.

Thegoogle.golang.org/protobuf/testing/protocmppackage provides functions to compare protocol buffer messages with thegithub.com/google/cmppackage.

Thegoogle.golang.org/protobuf/compiler/protogenpackage provides support for writing protocol compiler plugins.

Conclusion

The google.golang.org/protobuf module is a major overhaul ofGo's support for protocol buffers, providing first-class supportfor reflection, custom message implementations, and a cleaned up APIsurface. We intend to maintain the previous API indefinitely as a wrapperof the new one, allowing users to adopt the new API incrementally attheir own pace.

Our goal in this update is to improve upon the benefits of the oldAPI while addressing its shortcomings. As we completed each component ofthe new implementation, we put it into use within Google's codebase. Thisincremental rollout has given us confidence in both the usability of the newAPI and the performance and correctness of the new implementation. We believeit is production ready.

We are excited about this release and hope that it will serve the Goecosystem for the next ten years and beyond!

Go always comes second to morebasic concerns like personal and family health and safety.Around the world, the past couple months have been terrible,and we are still at the start of this awful pandemic.There are days when it seems like working onanything related to Go should be considered a serious priority inversion.

But after we’ve done all we canto prepare ourselves and our families for whatever is coming,getting back to some approximation of a familiar routineand normal work is a helpful coping mechanism.In that spirit, we intend to keep working on Goand trying to help the Go community as much as we can.

In this post we want to share a few important notes abouthow the pandemic is affecting the Go community,a few things we’re doing to help, what you can do to help,and our plans for Go itself.

Conferences and Meetups

The Go community thrives on in-person conferences and meetups.We had anticipated 35 conferences this yearand thousands of meetups, nearly all of which havenow changed, been postponed, or been cancelled.We’ll keep theconferences wiki pageupdated as plans change.

We want to do everything we can to help support impacted Go conferences.We also want to support efforts to explorenew ways for gophers to connect in the time of social distancing.In addition to honoring Google’s existing sponsorships,we are interested to offer support to people planningvirtual conference alternatives through the rest of the year.If you are organizing a Go conference and have been impacted,or if you are considering holding a virtual alternative,please reach out to Carmen Andoh at candoh@google.com.

For conference organizers,the Gophers slack#conf-organizers channelis a place to discuss contingency plans,best practices, cancellation, and postponement support.It’s also a place to share idea for virtual events,to continue to connect and support the Go community.

For meetup organizers,the Go Developer Networkcan provide Zoom for Education licensing to meetupsthat want to start holding virtual meetings.If you host a meetup, or you’d like to, we encourage youto use this opportunity to get speakers from outside yourregion to present to your group.For more information, and to get involved,please jointhe Gophers slack#remotemeetup channel.

Online Training

The Go trainers you meet at conferences also travel the globe doingin-person trainingfor companies that want help adopting Go.That in-person teaching is crucial to bringingnew gophers into the community;we’re incredibly grateful to the trainers for the work they do.Unfortunately, on-site training contracts have all been cancelledfor the next few months, and the trainers in our communityhave lost their primary (or sole) source of income.We encourage companies to consider virtual trainingand workshops during this difficult time.Most trainers are being flexible with pricing,scheduling, and class structure.

Job Postings

We know that the current downturn means that somegophers are looking for new jobs.The Go community has built a number of Go-specific job-posting sites, includingGolang Cafe,Golang Projects,andWe Love Go.The Gophers slackalso has many job-hunting channels: search for “job” in the channel list.We encourage employers with any new openings to post in asmany appropriate places as possible.

FOSS Responders

We are proud that Go is part of the broader open-source ecosystem.FOSS Respondersis one effort to help the open-source ecosystemdeal with the impacts of the pandemic.If you want to do something to help affected open-source communities,they are coordinating efforts and also have links to other efforts.And if you know of other open-source communities that need help,let them know about FOSS Responders.

COVID-19 Open-Source Help Desk

The COVID-19 Open-Source Help Deskaims to help virologists, epidemiologists, and other domain expertsfind quick answers to any problems they are having withopen-source scientific computing software,from experts in that software,so they can focus their time on what they know best.If you are a developer or a scientific computing expertwilling to help by answering the posts of the domain experts,visit the site to learn how to help.

U.S. Digital Response

For our gophers in the United States,the U.S. Digital Responseis working to connect qualified volunteers tostate and local governments that need digital helpduring this crisis.Quoting the web page,“If you have relevant experience(healthcare, data, engineering & product development,general management, operations, supply chain/procurement and more),can work autonomously through ambiguity,and are ready to jump into a high-intensity environment,”see the site for how to volunteer.

Plans for Go

Here on the Go team at Google, we recognize that theworld around us is changing rapidlyand that plans beyond the next couple weeksare not much more than hopeful guesses.That said, right now we are workingon what we think are the most important projects for 2020.Like all of you, we’re at reduced capacity, so the workcontinues slower than planned.

Our analysis of the Go 2019 user survey is almost complete,and we hope to post it soon.

At least for now, we intend to keep to our timeline for Go 1.15,with the understanding that it will probably have fewer new featuresand improvements than we originally planned.We continue to do code reviews, issue triage,and proposal review.

Goplsis the language-aware backend supporting most Go editors today,and we continue to work toward its 1.0 release.

The new Go package and module site pkg.go.devkeeps getting better.We’ve been working on usability improvementsand new features to better help users find and evaluate Go packages.We’ve also expanded the set of recognized licenses and improved thelicense detector, with more improvements to come.

Our Gopher valuesare what ground us, now more than ever.We are working extra hard to be friendly, welcoming,patient, thoughtful, respectful, and charitable.We hope everyone in the Go community will try to do the same.

We’ll continue to use this blog to let you know aboutimportant news for the Go ecosystem.In those moments when you’ve taken care of the much moreimportant things going on in your life,we hope you’ll check in and see what we’ve been up to.

Thank you, as always, for using Go and being part of the Go community.We wish you all the best in these difficult times.

What a response!

I want to start with an enormous thank you to the thousands of Go developerswho participated in this year’s survey.For 2019, we saw 10,975 responses, nearly twice as many as last year!On behalf of the rest of the team, I cannot adequately stress how much weappreciate you taking the time and effort to tell us about your experiences with Go. Thank you!

A note about prior years

Sharp-eyed readers may notice that our year-over-year comparisons don’tquite square with numbers we’ve shared in the past.The reason is that from 2016–2018, we calculated percentages for eachquestion using the total number of people who started the survey as the denominator.While that’s nice and consistent, it ignores the fact that not everyonefinishes the survey—up to 40% of participants stop before reaching the final page,which meant questions that occurred later in the survey appeared to performworse solely because they were later.Thus, this year we’ve recalculated all of our results (including the 2016–2018responses shown in this post) to use the number of people who respondedto a given question as the denominator for that question.We’ve included the number of 2019 responses for each chart—in the formof "n=[number of respondents]" on the x-axis or in the chart’s legend—togive readers a better understanding of the weight of evidence underlying each finding.

Similarly, we learned that in prior surveys options that appeared earlierin response lists had a disproportionate response rate.To address this, we added an element of randomization into the survey.Some of our multiple-choice questions have lists of choices with no logical ordering,such as "I write the following in Go:[list of types of applications]".Previously these choices had been alphabetized,but for 2019 they were presented in a random order to each participant.This means year-over-year comparison for certain questions are invalid for 2018 → 2019,but trends from 2016–2018 are not invalidated.You can think of this as setting a more accurate baseline for 2019.We retained alphabetical ordering in cases where respondents are likelyto scan for a particular name,such as their preferred editor.We explicitly call out which questions this applies to below.

A third major change was to improve our analysis of questions with open-ended,free-text responses.Last year we used machine learning to roughly—but quickly—categorize these responses.This year two researchers manually analyzed and categorized these responses,allowing for a more granular analysis but preventing valid comparisons withlast year’s numbers.Like the randomization discussed above, the purpose of this change is togive us a reliable baseline for 2019 onward.

Without further ado…

This is a long post. Here’s the tl;dr of our major findings:

• The demographics of our respondents are similar to Stack Overflow’s survey respondents,which increases our confidence that these results are representative ofthe larger Go developer audience.
• A majority of respondents use Go every day, and this number has been trending up each year.
• Go’s use is still concentrated in technology companies,but Go is increasingly found in a wider variety of industries,such as finance and media.
• Methodology changes showed us that most of our year-over-year metricsare stable and higher than we previously realized.
• Respondents are using Go to solve similar problems,particularly building API/RPC services and CLIs,regardless of the size of organization they work at.
• Most teams try to update to the latest Go release quickly;when third-party providers are late to support the current Go release,this creates an adoption blocker for developers.
• Almost everyone in the Go ecosystem is now using modules, but some confusion around package management remains.
• High-priority areas for improvement include improving the developer experience for debugging,working with modules, and working with cloud services.
• VS Code and GoLand have continued to see increased use; they’re now preferred by 3 out of 4 respondents.

Who did we hear from?

This year we asked some new demographic questions to help us better understandthe people who’ve responded to this survey.In particular, we asked about the duration of professional programming experienceand the size of the organizations where people work.These were modeled on questions that StackOverflow asks in their annual survey,and the distribution of responses we saw is very close to StackOverflow’s 2019 results.Our take-away is the respondents to this survey have similar levels of professionalexperience and proportional representation of different sizes of organizationsas the StackOverflow survey audience (with the obvious difference that we’reprimarily hearing from developers working with Go).That increases our confidence when generalizing these findings to the estimated1 million Go developers worldwide.These demographic questions will also help us in the future to identifywhich year-over-year changes may be the result of a shift in who responded to the survey,rather than changes in sentiment or behavior.

Looking at Go experience, we see that a majority of respondents (56%) arerelatively new to Go,having used it for less than two years.Majorities also said they use Go at work (72%) and outside of work (62%).The percentage of respondents using Go professionally appears to be trending up each year.

As you can see in the chart below, in 2018 we saw a spike in these numbers,but that increase disappeared this year.This is one of many signals suggesting that the audience who answered thesurvey in 2018 was significantly different than in the other three years.In this case they were significantly more likely to be using Go outsideof work and a different language while at work,but we see similar outliers across multiple survey questions.

Respondents who have been using Go the longest have different backgroundsthan newer Go developers.These Go veterans were more likely to claim expertise in C/C++ and lesslikely to claim expertise in JavaScript,TypeScript, and PHP.One caveat is that this is self-reported "expertise";it may be more helpful to think of it instead as "familiarity".Python appears to be the language (other than Go) familiar to the most respondents,regardless of how long they’ve been working with Go.

Last year we asked about which industries respondents work in,finding that a majority reported working in software,internet, or web service companies.This year it appears respondents represent a broader range of industries.However, we also simplified the list of industries to reduce confusion frompotentially overlapping categories (e.g.,the separate categories for "Software" and "Internet / web services" from2018 were combined into "Technology" for 2019).Thus, this isn’t strictly an apples-to-apples comparison.For example, it’s possible that one effect of simplifying the category listwas to reduce the use of the "Software" category as a catch-all for respondentswriting Go software for an industry that wasn’t explicitly listed.

Go is a successful open-source project, but that doesn’t mean the developersworking with it are also writing free or open-source software.As in prior years, we found that most respondents are not frequent contributorsto Go open-source projects,with 75% saying they do so "infrequently" or "never".As the Go community expands, we see the proportion of respondents who’venever contributed to Go open-source projects slowly trending up.

Developer tools

As in prior years, the vast majority of survey respondents reported workingwith Go on Linux and macOS systems.This is one area of strong divergence between our respondents and StackOverflow’s 2019 results:in our survey, only 20% of respondents use Windows as a primary development platform,while for StackOverflow it was 45% of respondents.Linux is used by 66% and macOS by 53%—both much higher than the StackOverflow audience,which reported 25% and 30%, respectively.

The trend in editor consolidation has continued this year.GoLand saw the sharpest increase in use this year,rising from 24% → 34%.VS Code’s growth slowed, but it remains the most popular editor among respondents at 41%.Combined, these two editors are now preferred by 3 out of 4 respondents.

Every other editor saw a small decrease. This doesn’t mean those editorsaren’t being used at all,but they’re not what respondents say they prefer to use for writing Go code.

This year we added a question about internal Go documentation tooling,such as gddo.A small minority of respondents (6%) reported that their organization runsits own Go documentation server,though this proportion nearly doubles (to 11%) when we look at respondentsat large organizations (those with at least 5,000 employees).A follow-up asked of respondents who said their organization had stoppedrunning its own documentation server suggests that the top reason to retiretheir server was a combination of low perceived benefits (23%) versus theamount of effort required to initially set it up and maintain it (38%).

Sentiments towards Go

Large majorities of respondents agreed that Go is working well for theirteams (86%) and that they’d prefer to use it for their next project (89%).We also found that over half of respondents (59%) believe Go is criticalto the success of their companies.All of these metrics have remained stable since 2016.

Normalizing the results changed most of these numbers for prior years.For example, the percentage of respondents who agreed with the statement"Go is working well for my team" was previously in the 50’s and 60’s becauseof participant drop-off;when we remove participants who never saw the question,we see it’s been fairly stable since 2016.

Looking at sentiments toward problem solving in the Go ecosystem,we see similar results.Large percentages of respondents agreed with each statement (82%–88%),and these rates have been largely stable over the past four years.

This year we took a more nuanced look at satisfaction across industriesto establish a baseline.Overall, respondents were positive about using Go at work,regardless of industry sector.We do see small variations in dissatisfaction in a few areas,most notably manufacturing, which we plan to investigate with follow-up research.Similarly, we asked about satisfaction with—and the importance of—variousaspects of Go development.Pairing these measures together highlighted three topics of particular focus:debugging (including debugging concurrency),using modules, and using cloud services.Each of these topics was rated "very" or "critically" important by a majorityof respondents but had significantly lower satisfaction scores compared to other topics.

Turning to sentiments toward the Go community,we see some differences from prior years.First, there is a dip in the percentage of respondents who agreed with thestatement "I feel welcome in the Go community", from 82% to 75%.Digging deeper revealed that the proportion of respondents who "slightly"or "moderately agreed" decreased,while the proportions who "neither agree nor disagree" and "strongly agree"both increased (up 5 and 7 points, respectively).This polarizing split suggests two or more groups whose experiences in theGo community are diverging,and is thus another area we plan to further investigate.

The other big differences are a clear upward trend in responses to the statement"I feel welcome to contribute to the Go project" and a large year-over-yearincrease in the proportion of respondents who feel Go’s project leadershipunderstands their needs.

All of these results show a pattern of higher agreement correlated withincreased Go experience,beginning at about two years.In other words, the longer a respondent has been using Go,the more likely they were to agree with each of these statements.

This likely comes as no surprise, but people who responded to the Go DeveloperSurvey tended to like Go.However, we also wanted to understand which other languages respondents enjoy working with.Most of these numbers have not significantly changed from prior years,with two exceptions:TypeScript (which has increased 10 points),and Rust (up 7 points).When we break these results down by duration of Go experience,we see the same pattern as we found for language expertise.In particular, Python is the language and ecosystem that Go developers aremost likely to also enjoy building with.

In 2018 we first asked the "Would you recommend…" Net Promoter Score (NPS) question,yielding a score of 61.This year our NPS result is a statistically unchanged 60 (67% "promoters"minus 7% "detractors").

Working with Go

Building API/RPC services (71%) and CLIs (62%) remain the most common uses of Go.The chart below appears to show major changes from 2018,but these are most likely the result of randomizing the order of choices,which used to be listed alphabetically:3 of the 4 choices beginning with ’A’ decreased,while everything else remained stable or increased.Thus, this chart is best interpreted as a more accurate baseline for 2019with trends from 2016–2018.For example, we believe that the proportion of respondents building webservices which return HTML has been decreasing since 2016 but were likelyundercounted because this response was always at the bottom of a long list of choices.We also broke this out by organization size and industry but found no significant differences:it appears respondents use Go in roughly similar ways whether they workat a small tech start-up or a large retail enterprise.

A related question asked about the larger areas in which respondents work with Go.The most common area by far was web development (66%),but other common areas included databases (45%),network programming (42%), systems programming (38%),and DevOps tasks (37%).

In addition to what respondents are building,we also asked about some of the development techniques they use.A large majority of respondents said they depend upon text logs for debugging (88%),and their free-text responses suggest this is because alternative toolingis challenging to use effectively.However, local stepwise debugging (e.g., with Delve),profiling, and testing with the race detector were not uncommon,with ~50% of respondents depending upon at least one of these techniques.

Regarding package management, we found that the vast majority of respondentshave adopted modules for Go (89%).This has been a big shift for developers,and nearly the entire community appears to be going through it simultaneously.

We also found that 75% of respondents evaluate the current Go release for production use,with an additional 12% waiting one release cycle.This suggests a large majority of Go developers are using (or at the least,trying to use) the current or previous stable release,highlighting the importance for platform-as-a-service providers to quicklysupport new stable releases of Go.

Go in the clouds

Go was designed with modern distributed computing in mind,and we want to continue to improve the developer experience of buildingcloud services with Go.This year we expanded the questions we asked about cloud development tobetter understand how respondents are working with cloud providers,what they like about the current developer experience,and what can be improved.As mentioned earlier, some of the 2018 results appear to be outliers,such as an unexpectedly low result for self-owned servers,and an unexpectedly high result for GCP deployments.

We see two clear trends:

1. The three largest global cloud providers (Amazon Web Services,Google Cloud Platform, and Microsoft Azure) all appear to be trending upin usage among survey respondents,while most other providers are used by a smaller proportion of respondents each year.
2. On-prem deployments to self-owned or company-owned servers continue todecrease and are now statistically tied with AWS (44% vs.42%) as the most common deployment targets.

Looking at which types of cloud platforms respondents are using,we see differences between the major providers.Respondents deploying to AWS and Azure were most likely to be using VMsdirectly (65% and 51%,respectively), while those deploying to GCP were almost twice as likelyto be using the managed Kubernetes platform (GKE,64%) than VMs (35%).We also found that respondents deploying to AWS were equally likely to beusing a managed Kubernetes platform (32%) as they were to be using a managedserverless platform (AWS Lambda, 33%).Both GCP (17%) and Azure (7%) had lower proportions of respondents usingserverless platforms,and free-text responses suggest a primary reason was delayed support forthe latest Go runtime on these platforms.

Overall, a majority of respondents were satisfied with using Go on all threemajor cloud providers.Respondents reported similar satisfaction levels with Go development forAWS (80% satisfied) and GCP (78%).Azure received a lower satisfaction score (57% satisfied),and free-text responses suggest that the main driver was a perception thatGo lacks first-class support on this platform (25% of free-text responses).Here, "first-class support" refers to always staying up-to-date with the latest Go release,and ensuring new features are available to Go developers at time of launch.This was the same top pain-point reported by respondents using GCP (14%),and particularly focused on support for the latest Go runtime in serverless deployments.Respondents deploying to AWS, in contrast,were most likely to say the SDK could use improvements,such as being more idiomatic (21%).SDK improvements were also the second most common request for both GCP (9%)and Azure (18%) developers.

Pain points

The top reasons respondents say they are unable to use Go more remain workingon a project in another language (56%),working on a team that prefers to use another language (37%),and the lack of a critical feature in Go itself (25%).

This was one of the questions where we randomized the choice list,so year-over-year comparisons aren’t valid,though 2016–2018 trends are.For example, we are confident that the number of developers unable to useGo more frequently because their team prefers a different language is decreasing each year,but we don’t know whether that decrease dramatically accelerated this year,or was always a bit lower than our 2016–2018 numbers estimated.

The top two adoption blockers (working on an existing non-Go project andworking on a team that prefers a different language) don’t have direct technical solutions,but the remaining blockers might.Thus, this year we asked for more details,to better understand how we might help developers increase their use of Go.The charts in the remainder of this section are based on free-text responseswhich were manually categorized,so they have very long tails;categories totalling less than 3% of the total responses have been groupedinto the "Other" category for each chart.A single response may mention multiple topics,thus charts do not not sum to 100%.

Among the 25% of respondents who said Go lacks language features they need,79% pointed to generics as a critical missing feature.Continued improvements to error handling (in addition to the Go 1.13 changes) was cited by 22%,while 13% requested more functional programming features,particularly built-in map/filter/reduce functionality.To be clear, these numbers are from the subset of respondents who said theywould be able to use Go more were it not missing one or more critical features they need,not the entire population of survey respondents.

Respondents who said Go "isn’t an appropriate language" for what they workon had a wide variety of reasons and use-cases.The most common was that they work on some form of front-end development (22%),such as GUIs for web, desktop, or mobile.Another common response was that the respondent said they worked in a domainwith an already-dominant language (9%),making it a challenge to use something different.Some respondents also told us which domain they were referring to (or simplymentioned a domain without mentioning another language being more common),which we show via the "I work on [domain]" rows below.An additional top reason cited by respondents was a need for better performance (9%),particularly for real-time computing.

The biggest challenges respondents reported remain largely consistent with last year.Go’s lack of generics and modules/package management still top the list(15% and 12% of responses,respectively), and the proportion of respondents highlighting tooling problems increased.These numbers are different from the above charts because this questionwas asked of all respondents,regardless of what they said their biggest Go adoption blockers were.All three of these are areas of focus for the Go team this year,and we hope to greatly improve the developer experience,particularly around modules, tooling, and the getting started experience,in the coming months.

Diagnosing faults and performance issues can be challenging in any language.Respondents told us their top challenge for both of these was not somethingspecific to Go’s implementation or tooling,but a more fundamental issue:a self-reported lack of knowledge, experience, or best practices.We hope to help address these knowledge gaps via documentation and othereducational materials later this year.The other major problems do involve tooling,specifically a perceived unfavorable cost/benefit trade-off to learning/usingGo’s debugging and profiling tooling,and challenges making the tooling work in various environments (e.g.,debugging in containers, or getting performance profiles from production systems).

Finally, when we asked what would most improve Go support in respondents’editing environment,the most common response was for general improvements or better supportfor the language server (gopls, 19%).This was expected, as gopls replaces about 80 extant tools and is still in beta.When respondents were more specific about what they’d like to see improved,they were most likely to report the debugging experience (14%) and fasteror more reliable code completion (13%).A number of participants also explicitly referenced the need to frequentlyrestart VS Code when using gopls (8%);in the time since this survey was in the field (late November – early December 2019),many of these gopls improvements have already landed,and this continues to be a high-priority area for the team.

The Go community

Roughly two thirds of respondents used Stack Overflow to answer their Go-related questions (64%).The other top sources of answers were godoc.org (47%),directly reading source code (42%), and golang.org (33%).

The long tail on the previous chart highlights the large variety of differentsources (nearly all of them community-driven) and modalities that respondentsrely on to overcome challenges while developing with Go.Indeed, for many Gophers, this may be one of their main points of interactionwith the larger community:as our community expands, we’ve seen higher and higher proportions of respondentswho do not attend any Go-related events.For 2019, that proportion nearly reached two thirds of respondents (62%).

Due to updated Google-wide privacy guidelines,we can no longer ask about which countries respondents live in.Instead we asked about preferred spoken/written language as a very roughproxy for Go’s worldwide usage,with the benefit of providing data for potential localization efforts.

Because this survey is in English, there is likely a strong bias towardEnglish speakers and people from areas where English is a common second or third language.Thus, the non-English numbers should be interpreted as likely minimums ratherthan an approximation of Go’s global audience.

We found 12% of respondents identify with a traditionally underrepresented group (e.g.,ethnicity, gender identity, et al.) and 3% identify as female.(This question should have said "woman" instead of "female".The mistake has been corrected in our draft survey for 2020,and we apologize for it.)We strongly suspect this 3% is undercounting women in the Go community.For example, we know women software developers in the US respond to theStackOverflow Developer Survey at about half the rate we’d expect based on US employment figures (11% vs 20%).Since we don’t know the proportion of responses in the US,we can’t safely extrapolate from these numbers beyond saying the actualproportion is likely higher than 3%.Furthermore, GDPR required us to change how we ask about sensitive information,which includes gender and traditionally underrepresented groups.Unfortunately these changes prevent us from being able to make valid comparisonsof these numbers with prior years.

Respondents who identified with underrepresented groups or preferred notto answer this question showed higher rates of disagreement with the statement"I feel welcome in the Go community" (8% vs.4%) than those who do not identify with an underrepresented group,highlighting the importance of our continued outreach efforts.

Conclusion

We hope you’ve enjoyed seeing the results of our 2019 developer survey.Understanding developers’ experiences and challenges helps us plan and prioritize work for 2020.Once again, an enormous thank you to everyone who contributed to this survey—yourfeedback is helping to steer Go’s direction in the coming year and beyond.

When the Go project began, “an overarching goal was that Go do more to help theworking programmer by enabling tooling, automating mundane tasks such as codeformatting, and removing obstacles to working on large code bases”(Go FAQ).Today, more than a decade later, we continue to be guided by that same goal,especially as it pertains to the programmer’s most critical tool: their editor.

Throughout the past decade, Go developers have relied on a variety of editorsand dozens of independently authored tools and plugins. Much of Go’s earlysuccess can be attributed to the fantastic development tools created by the Gocommunity. TheVS Code extension for Go, built usingmany of these tools, is now used by 41 percent of Go developers(Go developer survey).

As the VS Code Go extension grows in popularity and asthe ecosystem expands, itrequiresmore maintenance and support.Over the past few years, the Go team has collaborated with the VS Code team tohelp the Go extension maintainers. The Go team also began a new initiative toimprove the tools powering all Go editor extensions, with a focus on supportingtheLanguage Server Protocolwith gopls andthe Debug Adapter Protocol with Delve.

Through this collaborative work between the VS Code and Go teams, we realizedthat the Go team is uniquely positioned to evolve the Go development experiencealongside the Go language.

As a result, we’re happy to announce the next phase in the Go team’spartnership with the VS Code team: The VS Code extension for Go is officiallyjoining the Go project. With this come two critical changes:

1. The publisher of the plugin is shifting from "Microsoft" to "Go Team at Google".
2. The project’s repository is moving to join the rest of the Go project at https://github.com/golang/vscode-go.

We cannot overstate our gratitude to those who have helpedbuild and maintain this beloved extension. We know that innovative ideas andfeatures come from you, our users. The Go team’s primary aim as owners of theextension is to reduce the burden of maintenance work on the Go community.We’ll make sure the builds stay green, the issues get triaged, and the docs getupdated. Go team members will keep contributors abreast of relevant languagechanges, and we’ll smooth the rough edges between the extension’s differentdependencies.

Please continue to share your thoughts with us by filingissues and makingcontributionsto the project. The process for contributing will now be the same as for therest of the Go project. Go teammembers will offer general help in the #vscode channel onGophers Slack, and we’ve also createda #vscode-dev channel to discuss issues and brainstorm ideas with contributors.

We’re excited about this new step forward, and we hope you are too.By maintaining a major Go editor extension, as well as the Go tooling andlanguage, the Go team will be able to provide all Go users, regardless of theireditor, a more cohesive and refined development experience.

As always, our goal remains the same: Every user should have an excellentexperience writing Go code.

See the accompanying post from the Visual Studio Code team.

We’re excited to announce that the codebase forpkg.go.dev is now open source.

The repository lives atgo.googlesource.com/pkgsiteand is mirrored togithub.com/golang/pkgsite.We will continue using the Go issue tracker to trackfeedbackrelated to pkg.go.dev.

Contributing

If you are interested in contributing to anyissues related to pkg.go.dev,check out ourcontribution guidelines.We also encourage you to continuefiling issuesif you run into problems or have feedback.

What’s Next

We really appreciate all the feedback we’ve received so far. It has been a bighelp in shaping ourroadmap for the coming year.Now that pkg.go.dev is open source, here’s what we’ll be working on next:

• We have some design changes planned for pkg.go.dev,to addressUX feedbackthat we have received. You can expect a more cohesive search and navigationexperience. We plan to share these designs for feedback once they are ready.

• We know that there are features available on godoc.org that userswant to see on pkg.go.dev. We’ve been keeping track of them onGo issue #39144,and will prioritize adding them in the next few months. We also plan tocontinue improving our license detection algorithm based on feedback.

• We’ll be improving our search experience based on feedback inGo issue #37810,to make it easier for users to find the dependencies they are looking for andmake better decisions around which ones to import.

Thanks for being patient with us in the process of open sourcing pkg.go.dev.We’re looking forward to receiving your contributions and working with you onthe future of the project.

Introduction

It’s been almost a year since we last wrote about the possibility ofadding generics to Go.It’s time for an update.

Updated design

We’ve been continuing to refine the generics designdraft.We’ve written a type checker for it: a program that can parse Go codethat uses generics as described in the design draft and report anytype errors.We’ve written example code.And we’ve collected feedback from many, many people—thanks forproviding it!

Based on what we’ve learned, we’re releasing an updated designdraft.The biggest change is that we are dropping the idea of contracts.The difference between contracts and interface types was confusing, sowe’re eliminating that difference.Type parameters are now constrained by interface types.Interface types are now permitted to include type lists, though onlywhen used as constraints; in the previous design draft type lists werea feature of contracts.More complex cases will use a parameterized interface type.

We hope that people will find this design draft simpler and easier tounderstand.

Experimentation tool

To help decide how to further refine the design draft, we arereleasing a translation tool.This is a tool that permits people to type check and run code writtenusing the version of generics described in the design draft.It works by translating generic code into ordinary Go code.This translation process imposes some limitations, but we hope that itwill be good enough for people to get a feel for what generic Go codemight look like.The real implementation of generics, if they are accepted into thelanguage, will work differently.(We have only just begun to sketch out what a direct compilerimplementation would look like.)

The tool is available on a variant of the Go playground athttps://go2goplay.golang.org.This playground works just like the usual Go playground, but itsupports generic code.

You can also build and use the tool yourself.It is available in a branch of the master Go repo.Follow the instructions on installing Go fromsource.Where those instructions direct you to check out the latest releasetag, instead run git checkout dev.go2go.Then build the Go toolchain as directed.

The translation tool is documented inREADME.go2go.

Next steps

We hope that the tool will give the Go community a chance toexperiment with generics.There are two main things that we hope to learn.

First, does generic code make sense?Does it feel like Go?What surprises do people encounter?Are the error messages useful?

Second, we know that many people have said that Go needs generics, butwe don’t necessarily know exactly what that means.Does this draft design address the problem in a useful way?If there is a problem that makes you think “I could solve this if Gohad generics,” can you solve the problem when using this tool?

We will use the feedback we gather from the Go community to decide howto move forward.If the draft design is well received and doesn’t need significantchanges, the next step would be a formal language changeproposal.To set expectations, if everybody is completely happy with the designdraft and it does not require any further adjustments, the earliestthat generics could be added to Go would be the Go 1.17 release,scheduled for August 2021.In reality, of course, there may be unforeseen problems, so this is anoptimistic timeline; we can’t make any definite prediction.

Feedback

The best way to provide feedback for the language changes will be onthe mailing list golang-nuts@googlegroups.com.Mailing lists are imperfect, but they seem like our best option forinitial discussion.When writing about the design draft, please put [generics] at thestart of the Subject line and to start different threads for differentspecific topics.

If you find bugs in the generics type checker or the translation tool,they should be filed in the standard Go issue tracker athttps://golang.org/issue.Please start the issue title with cmd/go2go:.Note that the issue tracker is not the best place to discuss changesto the language, because it does not provide threading and it is notwell suited to lengthy conversations.

We look forward to your feedback.

Acknowledgements

We’re not finished, but we’ve come a long way.We would not be here without a lot of help.

We’d like to thank Philip Wadler and his collaborators for thinkingformally about generics in Go and helping us clarify the theoreticalaspects of the design.Their paper Featherweight Goanalyzes generics in a restricted version of Go, and they havedeveloped a prototype on GitHub.

We would also like to thank thepeoplewho provided detailed feedback on an earlier version of the designdraft.

And last but definitely not least, we’d like to thank many people onthe Go team, many contributors to the Go issue tracker, and everybodyelse who shared ideas and feedback on earlier design drafts.We read all of it, and we’re grateful. We wouldn’t be here withoutyou.

Introduction

This post is part 5 in a series.

• Part 1 — Using Go Modules
• Part 2 — Migrating To Go Modules
• Part 3 — Publishing Go Modules
• Part 4 — Go Modules: v2 and Beyond
• Part 5 — Keeping Your Modules Compatible (this post)

Your modules will evolve over time as you add new features, change behaviors, and reconsider parts of the module's public surface. As discussed in Go Modules: v2 and Beyond, breaking changes to a v1+ module must happen as part of a major version bump (or by adopting a new module path).

However, releasing a new major version is hard on your users. They have to find the new version, learn a new API, and change their code. And some users may never update, meaning you have to maintain two versions for your code forever. So it is usually better to change your existing package in a compatible way.

In this post, we'll explore some techniques for introducing non-breaking changes. The common theme is: add, don’t change or remove. We’ll also talk about how to design your API for compatibility from the outset.

Adding to a function

Often, breaking changes come in the form of new arguments to a function. We’ll describe some ways to deal with this sort of change, but first let’s look at a technique that doesn’t work.

When adding new arguments with sensible defaults, it’s tempting to add them as a variadic parameter. To extend the function

func Run(name string)

with an additional size argument which defaults to zero, one might propose

func Run(name string, size ...int)

on the grounds that all existing call sites will continue to work. While that is true, other uses of Run could break, like this one:

package mypkgvar runner func(string) = yourpkg.Run

The original Run function works here because its type is func(string), but the new Run function’s type is func(string, ...int), so the assignment fails at compile time.

This example illustrates that call compatibility is not enough for backward compatibility. There is, in fact, no backward-compatible change you can make to a function’s signature.

Instead of changing a function’s signature, add a new function. As an example, after the context package was introduced, it became common practice to pass a context.Context as the first argument to a function. However, stable APIs could not change an exported function to accept a context.Context because it would break all uses of that function.

Instead, new functions were added. For example, the database/sql package's Query method’s signature was (and still is)

func (db *DB) Query(query string, args ...interface{}) (*Rows, error)

When the context package was created, the Go team added a new method to database/sql:

func (db *DB) QueryContext(ctx context.Context, query string, args ...interface{}) (*Rows, error)

To avoid copying code, the old method calls the new one:

func (db *DB) Query(query string, args ...interface{}) (*Rows, error) {    return db.QueryContext(context.Background(), query, args...)}

Adding a method allows users to migrate to the new API at their own pace. Since the methods read similarly and sort together, and Context is in the name of the new method, this extension of the database/sql API did not degrade readability or comprehension of the package.

If you anticipate that a function may need more arguments in the future, you can plan ahead by making optional arguments a part of the function’s signature. The simplest way to do that is to add a single struct argument, as the crypto/tls.Dial function does:

func Dial(network, addr string, config *Config) (*Conn, error)

The TLS handshake conducted by Dial requires a network and address, but it has many other parameters with reasonable defaults. Passing a nil for config uses those defaults; passing a Config struct with some fields set will override the defaults for those fields. In the future, adding a new TLS configuration parameter only requires a new field on the Config struct, a change that is backward-compatible (almost always—see "Maintaining struct compatibility" below).

Sometimes the techniques of adding a new function and adding options can be combined by making the options struct a method receiver. Consider the evolution of the net package’s ability to listen at a network address. Prior to Go 1.11, the net package provided only a Listen function with the signature

func Listen(network, address string) (Listener, error)

For Go 1.11, two features were added to net listening: passing a context, and allowing the caller to provide a “control function” to adjust the raw connection after creation but before binding. The result could have been a new function that took a context, network, address and control function. Instead, the package authors added a ListenConfig struct in anticipation that more options might be needed someday. And rather than define a new top-level function with a cumbersome name, they added a Listen method to ListenConfig:

type ListenConfig struct {    Control func(network, address string, c syscall.RawConn) error}func (*ListenConfig) Listen(ctx context.Context, network, address string) (Listener, error)

Another way to provide new options in the future is the “Option types” pattern, where options are passed as variadic arguments, and each option is a function that changes the state of the value being constructed. They are described in more detail by Rob Pike's post Self-referential functions and the design of options. One widely used example is google.golang.org/grpc's DialOption.

Option types fulfill the same role as struct options in function arguments: they are an extensible way to pass behavior-modifying configuration. Deciding which to choose is largely a matter of style. Consider this simple usage of gRPC's DialOption option type:

grpc.Dial("some-target",  grpc.WithAuthority("some-authority"),  grpc.WithMaxDelay(time.Second),  grpc.WithBlock())

This could also have been implemented as a struct option:

notgrpc.Dial("some-target", &notgrpc.Options{  Authority: "some-authority",  MaxDelay:  time.Minute,  Block:     true,})

Functional options have some downsides: they require writing the package name before the option for each call; they increase the size of the package namespace; and it's unclear what the behavior should be if the same option is provided twice. On the other hand, functions which take option structs require a parameter which might almost always be nil, which some find unattractive. And when a type’s zero value has a valid meaning, it is clumsy to specify that the option should have its default value, typically requiring a pointer or an additional boolean field.

Either one is a reasonable choice for ensuring future extensibility of your module's public API.

Working with interfaces

Sometimes, new features require changes to publicly-exposed interfaces: for example, an interface needs to be extended with new methods. Directly adding to an interface is a breaking change, though—how, then, can we support new methods on a publicly-exposed interface?

The basic idea is to define a new interface with the new method, and then wherever the old interface is used, dynamically check whether the provided type is the older type or the newer type.

Let's illustrate this with an example from the archive/tar package. tar.NewReader accepts an io.Reader, but over time the Go team realized that it would be more efficient to skip from one file header to the next if you could call Seek. But, they could not add a Seek method to io.Reader: that would break all implementers of io.Reader.

Another ruled-out option was to change tar.NewReader to accept io.ReadSeeker rather than io.Reader, since it supports both io.Reader methods and Seek (by way of io.Seeker). But, as we saw above, changing a function signature is also a breaking change.

So, they decided to keep tar.NewReader signature unchanged, but type check for (and support) io.Seeker in tar.Reader methods:

package tartype Reader struct {  r io.Reader}func NewReader(r io.Reader) *Reader {  return &Reader{r: r}}func (r *Reader) Read(b []byte) (int, error) {  if rs, ok := r.r.(io.Seeker); ok {    // Use more efficient rs.Seek.  }  // Use less efficient r.r.Read.}

(See reader.go for the actual code.)

When you run into a case where you want to add a method to an existing interface, you may be able to follow this strategy. Start by creating a new interface with your new method, or identify an existing interface with the new method. Next, identify the relevant functions that need to support it, type check for the second interface, and add code that uses it.

This strategy only works when the old interface without the new method can still be supported, limiting the future extensibility of your module.

Where possible, it is better to avoid this class of problem entirely. When designing constructors, for example, prefer to return concrete types. Working with concrete types allows you to add methods in the future without breaking users, unlike interfaces. That property allows your module to be extended more easily in the future.

Tip: if you do need to use an interface but don't intend for users to implement it, you can add an unexported method. This prevents types defined outside your package from satisfying your interface without embedding, freeing you to add methods later without breaking user implementations. For example, see testing.TB's private() function.

type TB interface {    Error(args ...interface{})    Errorf(format string, args ...interface{})    // ...    // A private method to prevent users implementing the    // interface and so future additions to it will not    // violate Go 1 compatibility.    private()}

This topic is also explored in more detail in Jonathan Amsterdam's "Detecting Incompatible API Changes" talk (video, slides).

Add configuration methods

So far we've talked about overt breaking changes, where changing a type or a function would cause users' code to stop compiling. However, behavior changes can also break users, even if user code continues to compile. For example, many users expect json.Decoder to ignore fields in the JSON that are not in the argument struct. When the Go team wanted to return an error in that case, they had to be careful. Doing so without an opt-in mechanism would mean that the many users relying on those methods might start receiving errors where they hadn’t before.

So, rather than changing the behavior for all users, they added a configuration method to the Decoder struct: Decoder.DisallowUnknownFields. Calling this method opts a user in to the new behavior, but not doing so preserves the old behavior for existing users.

Maintaining struct compatibility

We saw above that any change to a function’s signature is a breaking change. The situation is much better with structs. If you have an exported struct type, you can almost always add a field or remove an unexported field without breaking compatibility. When adding a field, make sure that its zero value is meaningful and preserves the old behavior, so that existing code that doesn’t set the field continues to work.

Recall that the authors of the net package added ListenConfig in Go 1.11 because they thought more options might be forthcoming. Turns out they were right. In Go 1.13, the KeepAlive field was added to allow for disabling keep-alive or changing its period. The default value of zero preserves the original behavior of enabling keep-alive with a default period.

There is one subtle way a new field can break user code unexpectedly. If all the field types in a struct are comparable—meaning values of those types can be compared with == and != and used as a map key—then the overall struct type is comparable too. In this case, adding a new field of uncomparable type will make the overall struct type non-comparable, breaking any code that compares values of that struct type.

To keep a struct comparable, don’t add non-comparable fields to it. You can write a test for that, or rely on the upcoming gorelease tool to catch it.

To prevent comparison in the first place, make sure the struct has a non-comparable field. It may have one already—no slice, map or function type is comparable—but if not, one can be added like so:

type Point struct {        _ [0]func()        X int        Y int}

The func() type is not comparable, and the zero-length array takes up no space. We can define a type to clarify our intent:

type doNotCompare [0]func()type Point struct {        doNotCompare        X int        Y int}

Should you use doNotCompare in your structs? If you’ve defined the struct to be used as a pointer—that is, it has pointer methods and perhaps a NewXXX constructor function that returns a pointer—then adding a doNotCompare field is probably overkill. Users of a pointer type understand that each value of the type is distinct: that if they want to compare two values, they should compare the pointers.

If you are defining a struct intended to be used as a value directly, like our Point example, then quite often you want it to be comparable. In the uncommon case that you have a value struct that you don’t want compared, then adding a doNotCompare field will give you the freedom to change the struct later without having to worry about breaking comparisons. On the downside, the type won’t be usable as a map key.

Conclusion

When planning an API from scratch, consider carefully how extensible the API will be to new changes in the future. And when you do need to add new features, remember the rule: add, don't change or remove, keeping in mind the exceptions—interfaces, function arguments, and return values can't be added in backwards-compatible ways.

If you need to dramatically change an API, or if an API begins to lose its focus as more features are added, then it may be time for a new major version. But most of the time, making a backwards-compatible change is easy and avoids causing pain for your users.

Today the Go team is very happy to announce the release of Go 1.15. You can get it from the download page.

Some of the highlights include:

• Substantial improvements to the Go linker
• Improved allocation for small objects at high core counts
• X.509 CommonName deprecation
• GOPROXY supports skipping proxies that return errors
• New embedded tzdata package
• A number of Core Library improvements

For the complete list of changes and more information about the improvements above, see the Go 1.15 release notes.

We want to thank everyone who contributed to this release by writing code, filing bugs, providing feedback, and/or testing the beta and release candidates.Your contributions and diligence helped to ensure that Go 1.15 is as stable as possible.That said, if you notice any problems, please file an issue.

We hope you enjoy the new release!

Help shape the future of Go

Since 2016, thousands of Gophers around the world have helped the Go projectby sharing their thoughts via our annual Go Developer Survey.Your feedback has played an enormous role in driving changes to our language,ecosystem, and community, including the gopls language server,the latest generics draft, the module mirror, and so much more.And of course, we publicly share each year's results,so we can all benefit from the community's insights.

This year we’ve streamlined the survey to shorten the time it takes tocomplete and improved the survey’s accessibility.The specific questions each person will see are randomly selected,so folks who’ve taken the survey in prior years may not see all of thequestions they are used to.Rest assured that they are still there. This approach allows us to ask abouta wider variety of topics than in prior years while also reducing the survey’s length.

Today we are launching the 2020 Go Developer Survey.We'd love to hear from everyone who uses Go,used to use Go, or is interested in using Go,to help ensure the language, community, and ecosystem fit the needs of thepeople closest to it.Please help us shape Go's future by participating in this 10-minute survey by November 8th:Take the 2020 Go Developer Survey.

Spread the word!

We need as many Gophers as possible to participate in this survey to helpus better understand our global user base.We'd be grateful if you would spread the word by sharing this post on yoursocial network feeds,around the office, at virtual meet-ups, and in other communities. Thank you!

Since launching pkg.go.dev, we’ve received a lot of great feedback ondesign and usability.In particular, it was clear that the way information was organized confusedusers when navigating the site.

Today we’re excited to share a redesigned pkg.go.dev,which we hope will be clearer and more helpful.This blog post presents the highlights. For details,see Go issue 41585.

Consistent landing page for all paths

The main change is that the pkg.go.dev/<path> page has been reorganizedaround the idea of a path.A path represents a directory in a particular version of a module.Now, regardless of what’s in that directory,every path page will have the same layout,with the goal of making the experience consistently useful and predictable.

The path page will display the README at that path if there is one.Previously, the overview tab only showed the README if present at the module root.This is one of many changes we’re making to place the most important information up front.

Documentation navigation

The documentation section now displays an index along with a sidenav.This gives the ability to see the full package API,while having context as they are navigating the documentation section.There is also a new Jump To input box in the left sidenav,to search for identifiers.

See Go issue 41587 for details on changes in the documentation section.

Metadata on main page

The top bar on each page now shows additional metadata,such as each package’s “imports” and “imported by” counts.Banners also show information about the latest minor and major versions of a module.See Go issue 41588 for details.

Video Walkthrough

Last week at Google Open Source Live,we presented a walkthrough of the new site experience in our talk,Level Up: Go Package Discovery and Editor Tooling.

Feedback

We’re excited to share this updated design with you.As always, please let us know what you think via the “Share Feedback”and “Report an Issue” links at the bottom of every page of the site.

And if you’re interested in contributing to this project, pkg.go.dev is open source! Check out thecontribution guidelinesto find out more.

Today we celebrate the eleventh birthday of the Go open source release.The parties we had forGo turning 10seem like a distant memory.It’s been a tough year, butwe’ve kept Go development moving forwardand accumulated quite a few highlights.

In November, we launched go.dev and pkg.go.devshortly after Go’s 10th birthday.

In February, the Go 1.14 releasedelivered the first officially “production-ready” implementation of Go modules,along with many performance improvements,includingfaster defersandnon-cooperative goroutine preemptionto reduce schedulingand garbage collection latency.

In early March, we launched anew API for protocol buffers,google.golang.org/protobuf,with much-improved support for protocol buffer reflection and custom messages.

When the pandemic hit, we decided to pause any public announcementsor launches in the spring,recognizing that everyone’s attention rightly belonged elsewhere.But we kept working, and one of our team members joined theApple/Google collaboration onprivacy-preserving exposure notificationsto support contact tracing efforts all over the world.In May, that group launched thereference backend server,written in Go.

We continued to improve gopls,which enables advancedGo-aware supportin many editors.In June, theVSCode Go extension officially joined the Go projectand is now maintained by the same developers who work on gopls.

Also in June, thanks to your feedback, we open-sourcedthe code behind pkg.go.devas part of the Go project as well.

Later in June, wereleased the latest design draft for generics,along with a prototype tool and generics playground.

In July, we published and discussed three new design drafts for future changes:new //go:build lines for file selection,file system interfaces,andbuild-time file embedding.(We’ll see all of those in 2021, as noted below.)

In August, the Go 1.15 releasedelivered mainly optimizations and bug fixes rather than new features.The most significant was the start of a rewrite of the linker,making it run 20% faster and use 30% less memoryon average for large builds.

Last month, we ran our annual Go user survey.We will post results on the blog once we’ve analyzed them.

The Go community has adapted to “virtual-first” along with everyone else,and we saw many virtual meetups and over a dozen virtual Go conferences this year.Last week, the Go team hostedGo day at Google Open Source Live(videos at the link).

Going Forward

We’re also incredibly excited about what’s in store for Go’s 12th year.Most immediately, this week Go team members willbe presenting eight events atGopherCon 2020.Mark your calendars!

• “Typing [Generic] Go”,a talk by Robert Griesemer,
  Nov 11, 10:00 AM (US Eastern);Q&A at 10:30 AM.
• “What to Expect When You’re NOT Expecting”,a live taping of the Go time podcast with a panel of expert debuggers,including Hana Kim,
  Nov 11 12:00 PM.
• “Evolving the Go Memory Manager's RAM and CPU Efficiency”,a talk by Michael Knyszek,
  Nov 11 1:00 PM;Q&A at 1:50 PM.
• “Implementing Faster Defers”,a talk by Dan Scales,
  Nov 11 5:10 PM;Q&A at 5:40 PM.
• “Go Team - Ask Me Anything”,a live Q&A with Julie Qiu, Rebecca Stambler, Russ Cox, Sameer Ajmani, and Van Riper,
  Nov 12 3:00 PM.
• “Pardon the Interruption: Loop Preemption in Go 1.14”,a talk by Austin Clements,
  Nov 12 4:45 PM;Q&A at 5:15 PM.
• “Working with Errors”,a talk by Jonathan Amsterdam,
  Nov 13 1:00 PM;Q&A at 1:50 PM.
• “Crossing the Chasm for Go: Two Million Users and Growing”,a talk by Carmen Andoh,
  Nov 13 5:55 PM.

Go Releases

In February, the Go 1.16 release will include the newfile system interfacesandbuild-time file embedding.It will complete the linker rewrite, bringing additional performance improvements.And it will include support for the new Apple Silicon (GOARCH=arm64) Macs.

In August, the Go 1.17 release will no doubt bring more features and improvements,although it’s far enough out that the exact details remain up in the air.It will include a new register-based calling convention for x86-64(without breaking existing assembly!),which will make programs faster across the board.(Other architectures will follow in later releases.)One nice feature that will definitely be included is thenew //go:build lines,which are far less error-prone than thecurrent //+build lines.Another highly anticipated feature we hope will be ready for beta testing next yearissupport for fuzzing in the go test command.

Go Modules

Over the next year, we will continue to work on developing support for Go modulesand integrating them well into the entire Go ecosystem.Go 1.16 will include our smoothest Go modules experience yet.One preliminary result from our recent survey is that 96% of usershave now adopted Go modules (up from 90% a year ago).

We will also finally wind down support for GOPATH-based development:any programs using dependencies other than the standard library will need a go.mod.(If you haven’t switched to modules yet, see theGOPATH wiki pagefor details about this final step in the journey from GOPATH to modules.)

From the start, the goal for Go moduleshas been “to add the concept of package versions to the working vocabularyof both Go developers and our tools,”to enable deep support for modules and versions throughout the Go ecosystem.The Go module mirror, checksum database, and indexwere made possible by this ecosystem-wide understanding of what a package version is.Over the next year, we will see rich module support added to more tools and systems.For example, we plan to investigate new tooling to help module authors publish new versions(go release)as well as to help module consumers update their code to migrate away fromdeprecated APIs (a new go fix).

As a larger example,we created goplsto reduce many tools used by editors for Go support,none of which supported modules, down to a single one that did.Over the next year,we’ll be ready to make the VSCode Go extension use gopls by default,for an excellent module experience out of the box,and we’ll release gopls 1.0.Of course, one of the best things about gopls is that it is editor-neutral:any editor that understands thelanguage server protocolcan use it.

Another important use of version information is tracking whetherany package in a build has a known vulnerability.Over the next year, we plan to develop a database of known vulnerabilitiesas well as tools to check your programs against that database.

The Go package discovery sitepkg.go.devis another example of a version-aware system enabled by Go modules.We’ve been focused on getting the core functionality and user experience right,including aredesign launching today.Over the next year,we will be unifying godoc.org into pkg.go.dev.We will also be expanding the version timeline for each package,showing important changes in each version,known vulnerabilities, and more,following the overall goal of surfacing what you need to makeinformed decisions about adding dependencies.

We’re excited to see this journey from GOPATH to Go modulesnearing completion and all the excellent dependency-aware toolsthat Go modules are enabling.

Generics

The next feature on everyone’s minds is of course generics.As we mentioned above, we published thelatest design draft for genericsback in June.Since then, we’ve continued to refine rough edges and have turned ourattention to the details of implementing a production-ready version.We will be working on that throughout 2021, with a goal of havingsomething for people to try out by the end of the year,perhaps a part of the Go 1.18 betas.

Thank You!

Go is far more than just us on the Go team at Google.We are indebted to the contributors who work with us with the Go releases and tools.Beyond that, Go only succeeds because of all of you who work inand contribute to Go’s thriving ecosystem.It has been a difficult year in the world outside Go.More than ever, we appreciate you taking the timeto join us and help make Go such a success.Thank you.We hope you are all staying safe and wish you all the best.

With the introduction of Go modules and the growth of the Go ecosystem,pkg.go.dev waslaunched in 2019 to provide a central placewhere developers can discover and evaluate Go packages and modules. Likegodoc.org, pkg.go.dev serves Go documentation, but it also supports modules,better search functionality, and signals to help Go users to find the rightpackages.

As we shared in January 2020, ourgoal is to eventually redirect traffic from godoc.org to the corresponding pageon pkg.go.dev. We’ve also made it possible for users to opt in to redirectingtheir own requests from godoc.org to pkg.go.dev.

We’ve received a lot of great feedback this year, which has been tracked andresolved through thepkgsite/godoc.org-redirectand pkgsite/design-2020milestones on the Go issue tracker. Your feedback resulted in support forpopular feature requests on pkg.go.dev,open sourcing pkgsite, and most recently, aredesign of pkg.go.dev.

Next Steps

The next step in this migration is to redirect all requests from godoc.org tothe corresponding page on pkg.go.dev.

This will happen in early 2021, once the work tracked at thepkgsite/godoc.org-redirect milestoneis complete.

During this migration, updates will be posted toGo issue 43178.

We encourage everyone to begin using pkg.go.dev today. You can do so byvisiting godoc.org?redirect=on, or clicking“Always use pkg.go.dev” in the top right corner of any godoc.org page.

FAQs

Will godoc.org URLs continue to work?

Yes! We will redirect all requests arriving at godoc.org to the equivalent pageon pkg.go.dev, so all your bookmarks and links will continue to take you to thedocumentation you need.

What will happen to the golang/gddo repository?

The gddo repository will remain availablefor anyone who wants to keep running it themselves, or even fork and improveit. We will mark it archived to make clear that we will no longer acceptcontributions. However, you will be able to continue forking the repository.

Will api.godoc.org continue to work?

This transition will have no impact on api.godoc.org. Until an API is availablefor pkg.go.dev, api.godoc.org will continue to serve traffic. SeeGo issue 36785 for updates on an API forpkg.go.dev.

Will my godoc.org badges keep working?

Yes! Badge URLs will redirect to the equivalent URL on pkg.go.dev too. Yourpage will automatically get a new pkg.go.dev badge. You can also generate a newbadge at pkg.go.dev/badge if you would like toupdate your badge link.

Feedback

As always, feel free to file an issueon the Go issue tracker for any feedback.

Contributing

Pkg.go.dev is an open source project. Ifyou’re interested in contributing to the pkgsite project, check out thecontribution guidelinesor join the#pkgsite channel on Gophers Slackto learn more.

The industry is abuzz about non-x86 processors recently,so we thought it would be worth a brief post about Go’s support for them.

It has always been important to us for Go to be portable,not overfitting to any particular operating system or architecture.The initial open source release of Goincluded support for two operating systems (Linux and Mac OS X) and threearchitectures (64-bit x86,32-bit x86, and 32-bit ARM).

Over the years, we’ve added support for many more operating systems and architecture combinations:

• Go 1 (March 2012) supported the original systems as well as FreeBSD,NetBSD, and OpenBSD on 64-bit and 32-bit x86,and Plan 9 on 32-bit x86.
• Go 1.3 (June 2014) added support for Solaris on 64-bit x86.
• Go 1.4 (December 2014) added support for Android on 32-bit ARM and Plan 9 on 64-bit x86.
• Go 1.5 (August 2015) added support for Linux on 64-bit ARM and 64-bit PowerPC,as well as iOS on 32-bit and 64-bit ARM.
• Go 1.6 (February 2016) added support for Linux on 64-bit MIPS,as well as Android on 32-bit x86.It also added an official binary download for Linux on 32-bit ARM,primarily for Raspberry Pi systems.
• Go 1.7 (August 2016) added support for Linux on z Systems (S390x) and Plan 9 on 32-bit ARM.
• Go 1.8 (February 2017) added support for Linux on 32-bit MIPS,and it added official binary downloads for Linux on 64-bit PowerPC and z Systems.
• Go 1.9 (August 2017) added official binary downloads for Linux on 64-bit ARM.
• Go 1.12 (February 2018) added support for Windows 10 IoT Core on 32-bit ARM,such as the Raspberry Pi 3.It also added support for AIX on 64-bit PowerPC.
• Go 1.14 (February 2019) added support for Linux on 64-bit RISC-V.

Although the x86-64 port got most of the attention in the early days of Go,today all our target architectures are well supported by our SSA-based compiler back endand produce excellent code.We’ve been helped along the way by many contributors,including engineers from Amazon, ARM, Atos,IBM, Intel, and MIPS.

Go supports cross-compiling for all these systems out of the box with minimal effort.For example, to build an app for 32-bit x86-based Windows from a 64-bit Linux system:

GOARCH=386 GOOS=windows go build myapp  # writes myapp.exe

In the past year, several major vendors have made announcements of new ARM64hardware for servers,laptops and developer machines.Go was well-positioned for this. For years now,Go has been powering Docker, Kubernetes, and the rest of the Go ecosystemon ARM64 Linux servers,as well as mobile apps on ARM64 Android and iOS devices.

Since Apple’s announcement of the Mac transitioning to Apple silicon this summer,Apple and Google have been working together to ensure that Go and the broaderGo ecosystem work well on them,both running Go x86 binaries under Rosetta 2 and running native Go ARM64 binaries.Earlier this week, we released the first Go 1.16 beta,which includes native support for Macs using the M1 chip.You can download and try the Go 1.16 beta for M1 Macs and all your othersystems on the Go download page.(Of course, this is a beta release and, like all betas,it certainly has bugs we don’t know about.If you run into any problems, please report them at golang.org/issue/new.)

It’s always nice to use the same CPU architecture for local development as in production,to remove one variation between the two environments.If you deploy to ARM64 production servers,Go makes it easy to develop on ARM64 Linux and Mac systems too.But of course, it’s still as easy as ever to work on one system and cross-compilefor deployment to another,whether you’re working on an x86 system and deploying to ARM,working on Windows and deploying to Linux,or some other combination.

The next target we’d like to add support for is ARM64 Windows 10 systems.If you have expertise and would like to help,we’re coordinating work on golang.org/issue/36439.

Generics proposal

We’ve filed a Go language changeproposal to add support for typeparameters for types and functions, permitting a form of genericprogramming.

Why generics?

Generics can give us powerful building blocks that let us share codeand build programs more easily.Generic programming means writing functions and data structures wheresome types are left to be specified later.For example, you can write a function that operates on a slice of somearbitrary data type, where the actual data type is only specified whenthe function is called.Or, you can define a data structure that stores values of any type,where the actual type to be stored is specified when you create aninstance of the data structure.

Since Go was first released in 2009, support for generics has been oneof the most commonly requested language features.You can read more about why generics are useful inan earlier blog post.

Although generics have clear use cases, fitting them cleanly into alanguage like Go is a difficult task.One of the first (flawed) attempts to add generics toGo dates backall the way to 2010.There have been several others over the last decade.

For the last couple of years we’ve been working on a series of designdrafts that have culminated in a design based on typeparameters.This design draft has had a lot of input from the Go programmingcommunity, and many people have experimented with it using thegenerics playground described in anearlier blog post.Ian Lance Taylor gave a talk at GopherCon2019about why to add generics and the strategy we are now following.Robert Griesemer gave a follow-up talk about changes in the design,and the implementation, at GopherCon2020.The language changes are fully backward compatible, so existing Goprograms will continue to work exactly as they do today.We have reached the point where we think that the design draft is goodenough, and simple enough, to propose adding it to Go.

What happens now?

The language change proposal processis how we make changes to the Go language.We have now started this processto add generics to a future version of Go.We invite substantive criticisms and comments, but please try to avoidrepeating earlier comments, and please try to avoid simple plus-oneand minus-one comments.Instead, add thumbs-up/thumbs-down emoji reactions to comments withwhich you agree or disagree, or to the proposal as a whole.

As with all language change proposals, our goal is to drive toward aconsensus to either add generics to the language or let the proposaldrop.We understand that for a change of this magnitude it will beimpossible to make everybody in the Go community happy, but we intendto get to a decision that everybody is willing to accept.

If the proposal is accepted, our goal will be to have a complete,though perhaps not fully optimized, implementation for people to tryby the end of the year, perhaps as part of the Go 1.18 betas.

Today’s Go security releasefixes an issue involving PATH lookups in untrusted directoriesthat can lead to remote execution during the goget command.We expect people to have questions about what exactly this meansand whether they might have issues in their own programs.This post details the bug, the fixes we have applied,how to decide whether your own programs are vulnerable to similar problems,and what you can do if they are.

Go command & remote execution

One of the design goals for the go command is that most commands – includinggobuild, godoc, goget, goinstall, and golist– do not runarbitrary code downloaded from the internet.There are a few obvious exceptions:clearly gorun, gotest, and gogeneratedo run arbitrary code – that's their job.But the others must not, for a variety of reasons including reproducible builds and security.So when goget can be tricked into executing arbitrary code, we consider that a security bug.

If goget must not run arbitrary code, then unfortunately that meansall the programs it invokes, such as compilers and version control systems, are also inside the security perimeter.For example, we've had issues in the past in which clever use of obscure compiler featuresor remote execution bugs in version control systems became remote execution bugs in Go.(On that note, Go 1.16 aims to improve the situation by introducing a GOVCS settingthat allows configuration of exactly which version control systems are allowed and when.)

Today's bug, however, was entirely our fault, not a bug or obscure feature of gcc or git.The bug involves how Go and other programs find other executables,so we need to spend a little time looking at that before we can get to the details.

Commands and PATHs and Go

All operating systems have a concept of an executable path($PATH on Unix, %PATH% on Windows; for simplicity, we'll just use the term PATH),which is a list of directories.When you type a command into a shell prompt,the shell looks in each of the listed directories,in turn, for an executable with the name you typed.It runs the first one it finds, or it prints a message like “command not found.”

On Unix, this idea first appeared in Seventh Edition Unix's Bourne shell (1979). The manual explained:

The shell parameter $PATH defines the search path for the directory containing the command.Each alternative directory name is separated by a colon (:).The default path is :/bin:/usr/bin.If the command name contains a / then the search path is not used.Otherwise, each directory in the path is searched for an executable file.

Note the default: the current directory (denoted here by an empty string,but let's call it “dot”)is listed ahead of /bin and /usr/bin.MS-DOS and then Windows chose to hard-code that behavior:on those systems, dot is always searched first,automatically, before considering any directories listed in %PATH%.

As Grampp and Morris pointed out in theirclassic paper “UNIX Operating System Security” (1984),placing dot ahead of system directories in the PATHmeans that if you cd into a directory and run ls,you might get a malicious copy from that directoryinstead of the system utility.And if you can trick a system administrator to run ls in your home directorywhile logged in as root, then you can run any code you want.Because of this problem and others like it,essentially all modern Unix distributions set a new user's default PATHto exclude dot.But Windows systems continue to search dot first, no matter what PATH says.

For example, when you type the command

go version

on a typically-configured Unix,the shell runs a go executable from a system directory in your PATH.But when you type that command on Windows,cmd.exe checks dot first.If .\go.exe (or .\go.bat or many other choices) exists,cmd.exe runs that executable, not one from your PATH.

For Go, PATH searches are handled by exec.LookPath,called automatically byexec.Command.And to fit well into the host system, Go's exec.LookPathimplements the Unix rules on Unix and the Windows rules on Windows.For example, this command

out, err := exec.Command("go", "version").CombinedOutput()

behaves the same as typing goversion into the operating system shell.On Windows, it runs .\go.exe when that exists.

(It is worth noting that Windows PowerShell changed this behavior,dropping the implicit search of dot, but cmd.exe and theWindows C library SearchPath functioncontinue to behave as they always have.Go continues to match cmd.exe.)

The Bug

When goget downloads and builds a package that containsimport"C", it runs a program called cgo to prepare the Goequivalent of the relevant C code.The go command runs cgo in the directory containing the package sources.Once cgo has generated its Go output files,the go command itself invokes the Go compileron the generated Go filesand the host C compiler (gcc or clang)to build any C sources included with the package.All this works well.But where does the go command find the host C compiler?It looks in the PATH, of course. Luckily, while it runs the C compilerin the package source directory, it does the PATH lookupfrom the original directory where the go command was invoked:

cmd := exec.Command("gcc", "file.c")cmd.Dir = "badpkg"cmd.Run()

So even if badpkg\gcc.exe exists on a Windows system,this code snippet will not find it.The lookup that happens in exec.Command does not knowabout the badpkg directory.

The go command uses similar code to invoke cgo,and in that case there's not even a path lookup,because cgo always comes from GOROOT:

cmd := exec.Command(GOROOT+"/pkg/tool/"+GOOS_GOARCH+"/cgo", "file.go")cmd.Dir = "badpkg"cmd.Run()

This is even safer than the previous snippet:there's no chance of running any bad cgo.exe that may exist.

But it turns out that cgo itself also invokes the host C compiler,on some temporary files it creates, meaning it executes this code itself:

// running in cgo in badpkg dircmd := exec.Command("gcc", "tmpfile.c")cmd.Run()

Now, because cgo itself is running in badpkg,not in the directory where the go command was run,it will run badpkg\gcc.exe if that file exists,instead of finding the system gcc.

So an attacker can create a malicious package that uses cgo andincludes a gcc.exe, and then any Windows userthat runs goget to download and build the attacker's packagewill run the attacker-supplied gcc.exe in preference to anygcc in the system path.

Unix systems avoid the problem first because dot is typically notin the PATH and second because module unpacking does notset execute bits on the files it writes.But Unix users who have dot ahead of system directoriesin their PATH and are using GOPATH mode would be as susceptibleas Windows users.(If that describes you, today is a good day to remove dot from your pathand to start using Go modules.)

(Thanks to RyotaK for reporting this issue to us.)

The Fixes

It's obviously unacceptable for the goget command to downloadand run a malicious gcc.exe.But what's the actual mistake that allows that?And then what's the fix?

One possible answer is that the mistake is that cgo does the search for the host C compilerin the untrusted source directory instead of in the directory where the go commandwas invoked.If that's the mistake,then the fix is to change the go command to pass cgo the full path to thehost C compiler, so that cgo need not do a PATH lookup into the untrusted directory.

Another possible answer is that the mistake is to look in dotduring PATH lookups, whether happens automatically on Windowsor because of an explicit PATH entry on a Unix system.A user may want to look in dot to find a command they typedin a console or shell window,but it's unlikely they also want to look there to find a subprocess of a subprocessof a typed command.If that's the mistake,then the fix is to change the cgo command not to look in dot during a PATH lookup.

We decided both were mistakes, so we applied both fixes.The go command now passes the full host C compiler path to cgo.On top of that, cgo, go, and every other command in the Go distributionnow use a variant of the os/exec package that reports an error if it wouldhave previously used an executable from dot.The packages go/build and go/import use the same policy fortheir invocation of the go command and other tools.This should shut the door on any similar security problems that may be lurking.

Out of an abundance of caution, we also made a similar fix incommands like goimports and gopls,as well as the librariesgolang.org/x/tools/go/analysisandgolang.org/x/tools/go/packages,which invoke the go command as a subprocess.If you run these programs in untrusted directories –for example, if you gitcheckout untrusted repositoriesand cd into them and then run programs like these,and you use Windows or use Unix with dot in your PATH –then you should update your copies of these commands too.If the only untrusted directories on your computerare the ones in the module cache managed by goget,then you only need the new Go release.

After updating to the new Go release, you can update to the latest gopls by using:

GO111MODULE=on \go get golang.org/x/tools/gopls@v0.6.4

and you can update to the latest goimports or other tools by using:

GO111MODULE=on \go get golang.org/x/tools/cmd/goimports@v0.1.0

You can update programs that depend on golang.org/x/tools/go/packages,even before their authors do,by adding an explicit upgrade of the dependency during goget:

GO111MODULE=on \go get example.com/cmd/thecmd golang.org/x/tools@v0.1.0

For programs that use go/build, it is sufficient for you to recompile themusing the updated Go release.

Again, you only need to update these other programs if youare a Windows user or a Unix user with dot in the PATHand you run these programs in source directories you do not trustthat may contain malicious programs.

Are your own programs affected?

If you use exec.LookPath or exec.Command in your own programs,you only need to be concerned if you (or your users) run your programin a directory with untrusted contents.If so, then a subprocess could be started using an executablefrom dot instead of from a system directory.(Again, using an executable from dot happens always on Windowsand only with uncommon PATH settings on Unix.)

If you are concerned, then we've published the more restricted variantof os/exec as golang.org/x/sys/execabs.You can use it in your program by simply replacing

import "os/exec"

with

import exec "golang.org/x/sys/execabs"

and recompiling.

Securing os/exec by default

We have been discussing ongolang.org/issue/38736whether the Windows behavior of always preferring the current directoryin PATH lookups (during exec.Command and exec.LookPath)should be changed.The argument in favor of the change is that it closes the kinds ofsecurity problems discussed in this blog post.A supporting argument is that although the Windows SearchPath APIand cmd.exe still always search the current directory,PowerShell, the successor to cmd.exe, does not,an apparent recognition that the original behavior was a mistake.The argument against the change is that it could break existing Windowsprograms that intend to find programs in the current directory.We don’t know how many such programs exist,but they would get unexplained failures if the PATH lookupsstarted skipping the current directory entirely.

The approach we have taken in golang.org/x/sys/execabs maybe a reasonable middle ground.It finds the result of the old PATH lookup and then returns aclear error rather than use a result from the current directory.The error returned from exec.Command("prog") when prog.exe exists looks like:

prog resolves to executable in current directory (.\prog.exe)

For programs that do change behavior, this error should make very clear what has happened.Programs that intend to run a program from the current directory can useexec.Command("./prog") instead (that syntax works on all systems, even Windows).

We have filed this idea as a new proposal, golang.org/issue/43724.

We're happy to announce that the VS Code Go extension now enables the goplslanguage serverby default, to deliver more robust IDE features and better support for Gomodules.

(gopls provides IDE features, such as as intelligent autocompletion, signature help, refactoring, and workspace symbol search.)

When Go modules werereleased two years ago, they completely changed the landscape of Go developertooling. Tools like goimports and godef previously depended on the factthat code was stored in your $GOPATH. When the Go team began rewriting thesetools to work with modules, we immediately realized that we needed a moresystematic approach to bridge the gap.

As a result, we began working on a single Golanguage server,gopls, which provides IDE features, such as autocompletion, formatting, anddiagnostics to any compatible editor frontend. This persistent and unifiedserver is a fundamentalshift from the earliercollections of command-line tools.

In addition to working on gopls, we sought other ways of creating a stableecosystem of editor tooling. Last year, the Go team took responsibility for theGo extension for VS Code. As part of thiswork, we smoothed the extension’s integration with the language server—automatinggopls updates, rearranging and clarifying gopls settings, improving thetroubleshooting workflow, and soliciting feedback through a survey. We’ve alsocontinued to foster a community of active users and contributors who havehelped us improve the stability, performance, and user experience of the Goextension.

Announcement

January 28 marked a major milestone in both the gopls and VS Code Gojourneys, as gopls is now enabled by default in the Go extension for VS Code.

In advance of this switch we spent a long time iterating on the design, featureset, and user experience of gopls, focusing on improving performance andstability. For more than a year, gopls has been the default in most plugins forVim, Emacs, and other editors. We’ve had 24 gopls releases, and we’reincredibly grateful to our users for consistently providing feedback andreporting issues on each and every one.

We’ve also dedicated time to smoothing the new user experience. We hope that VSCode Go with gopls will be intuitive with clear error messages, but if you havea question or need to adjust some configuration, you’ll be able to find answersin our updated documentation.We have also recorded a screencastto help you get started, as well asanimationsto show off some hard-to-find features.

Gopls is the best way of working with Go code, especially with Go modules.With the upcoming arrival of Go 1.16, in which modules are enabled by default,VS Code Go users will have the best possible experience out-of-the-box.

Still, this switch does not mean that gopls is complete. We will continueworking on bug fixes, new features, and general stability. Our next area offocus will be improving the user experience when working with multiplemodules.Feedback from our larger user base will help inform our next steps.

So, what should you do?

If you use VS Code, you don’t need to do anything.When you get the next VS Code Go update, gopls will be enabled automatically.

If you use another editor, you are likely using gopls already. If not, seethe gopls user guideto learn how to enable gopls in your preferred editor. The Language ServerProtocol ensures that gopls will continue to offer the same features to everyeditor.

If gopls is not working for you, please see our detailed troubleshootingguideand file an issue. If you need to, you can always disable gopls in VSCode.

Thank you

To our existing users, thank you for bearing with us as we rewrote our cachinglayer for the third time. To our new users, we look forward to hearing yourexperience reports and feedback.

Finally, no discussion of Go tooling is complete without mentioning thevaluable contributions of the Go tools community. Thank you for the lengthydiscussions, detailed bug reports, integration tests, and most importantly,thank you for the fantastic contributions. The most exciting gopls featurescome from our passionate open-source contributors, and we are appreciative ofyour hard work and dedication.

Learn more

Watch the screencast for awalk-through of how to get started with gopls and VS Code Go, and see theVS Code Go READMEfor additional information.

If you’d like to read about gopls in more detail, see thegopls README.

Today the Go team is very happy to announce the release of Go 1.16.You can get it from the download page.

The newembed packageprovides access to files embedded at compile time using the new //go:embed directive.Now it is easy to bundle supporting data files into your Go programs,making developing with Go even smoother.You can get started using theembed package documentation.Carl Johnson has also written a nice tutorial,“How to use Go embed”.

Go 1.16 also addsmacOS ARM64 support(also known as Apple silicon).Since Apple’s announcement of their new arm64 architecture, we have been working closely with them to ensure Go is fully supported; see our blog post“Go on ARM and Beyond”for more.

Note that Go 1.16requires use of Go modules by default,now that, according to our 2020 Go Developer Survey,96% of Go developers have made the switch.We recently added official documentation for developing and publishing modules.

Finally, there are many other improvements and bug fixes,including builds that are up to 25% faster and use as much as 15% less memory.For the complete list of changes and more information about the improvements above,see theGo 1.16 release notes.

We want to thank everyone who contributed to this release by writing codefiling bugs, providing feedback, and testing the beta and release candidate.

Your contributions and diligence helped to ensure that Go 1.16 is as stable as possible.That said, if you notice any problems, pleasefile an issue.

We hope you enjoy the new release!