From the course: Databricks Certified Data Engineer Associate Cert Prep: 2 ELT with Spark SQL and Python

Set up IntelliJ for Databricks with Go

- [Instructor] One of the emerging standards here for doing SDK work for data engineers is the Go language and I'm going to take a look at how to do that inside of this IntelliJ environment. So you can see here I've got a Databricks certified data engineer project and inside of here I'm going to create my Go SDK code for Databricks. So the first thing that I will do is I will actually go ahead and make a new directory called gohello. And I'm going to CD into that directory. Now the first thing that we need to do inside of this directory is type in go mod init sample. And this will create this initial Go module here. And this Go module is going to hold all of the things that we need in terms of dependencies. So the first thing that I will run is the Databricks SDK. So we'll say go mod edit -require and we'll go ahead and load that in. And now you can see here that we've got this new SDK loaded. We also want to go through here and create a new project file as well. So I'll need to go ahead and touch that. So we'll say touch main.go. And this main.go will be empty to start with, but what I'm going to do is just paste in some code and get it running. So this is from the official documentation. You can see we have a function here and inside the imports we have a couple of imports. So inside the main function here is where we're doing all the work. So first up, there is a client that's created and then this will unpack what's coming back from the client. In this case it'll be a clusters list. So let's list all of the clusters that are available and then we'll say if there's an error right here, go ahead and panic, otherwise, let's go ahead and print out, do a for loop over all of the clusters that come back from the SDK. So pretty straightforward here and all we need to do is install all of the missing dependencies. So we'll type in go mod tidy. Perfect. And then we'll go ahead and grab copies of all the packages needed to support builds and test the packages. So we'll type 'em in the Go Mod vendor, so go mod vendor. Great. And now all we need to do is go ahead and get this thing running by typing go run main.go. And this is going to go through here and compile this and run it. And we should see a list of all the clusters. There we go. So pretty straightforward actually to get Go running. There's some great advantages of Go language. There's strong support for the Databricks SDK as well. So if you want to give somebody in your team a binary that just ships to let's say Kubernetes or ships to some serverless platform that talks to Databricks, this is a pretty good solution.

Contents