Support for selecting scala version (#544)

* WIP compile using worker supplied from provider

* Provider[ScalaWorker] contains (will contain) everything
  needed to run a specific version of the scala compiler
* Manually create ScalaWorker for 2.11/2.12

* Add scalareflect + scalacompiler to [ScalaWorker]

* Run scala_repl using compiler from [ScalaWorker]

* All rules using [ScalaWorker]

* run scala_binary, scala_macro_library,
  scala_test, scala_junit_test using [ScalaWorker]
* Tests now working with 2.12

* Create [ScalaWorker] using new_scala_repository

* Generate build file for scalac_2_12 in repository_rule

* Use filegroup for scalac worker files

* Specify scala version and label in new_scala_repository

* Download scala version specified as parameter
* Set label for ScalaWorker as @{name}//:{name}

* Run formatter

* Run scalatest for both 2.11/2.12

* Add initial [ScalaWorker] for both 2.11/2.12
* import scalatest + scalactic for 2.12
* Add 2.12 versions of scalatest_runner and scalatest_reporter
* Manually specify scalatest_reporter for scala_test rule

* Update scalatest to 3.0.5 + remove defaults from ScalaWorker

- rename scalac -> scalac_2_11

* Symlink scalac_worker sources to repo_rule workspace

* Resolve scalatest_reporter version in rule

- Pass both 2.11/2.12 versions as implicit attrs
- Add "major_version" to [ScalaWorker]

* Use new_scala_repository for default 2.11/2.12

* scala_repositories takes scala versions as parameters

* improve naming

* Use single scala version

* Fix runfiles not being generated

* Support multiple scala versions for tut and scrooge

* Format .bzl and BUILD files

* Tests passing with 2.11

* Make scala_proto work with 2.12

* Support specs2 for both 2.11/2.12

- remove unused attr major_version from ScalacProvider

* Don't pass scalatest dependencies through provider

* Pass _scalac through _implicit_deps

* Remove temp test directory

* Remove debugging println

* Cleanup

* Run formatter

* Use inputs from resolve_command

* Combine scalatest+scalactic to single target

* Add utils

- extract_major_version_underscore()
- default_scala_version()
- run formatter

* Increase scala_mvn_artifact usage, add shas

* Add default_major_scala_version()

* Add scala_jar_shas dict

- Specify only one version of scalatest/scalactic...
  getting the shas from scala_jar_shas

* Fix aspect test

* Use scala_maven_import_external for all imports

specs2, tut, scrooge, protob

* Add util_core to scrooge, fix aspect test

* Supply shas to scala_library,compiler and reflect

- User can add shas for custom scala versions
- run formatter

* namespace scala_default

* Scala default version to 2.11.12

- correct spelling error

* Lookup scala_extra_jar_shas[major_version] once, renaming

* tut server_urls as parameter

* Pass ScalacProvider as paramater

* Replace pass classpaths instead of jars

- Stop using ScalacProvider.scala_library etc.
- Use ScalacProvider.default_compile_classpath etc.

* linting

* Lookup scala_jar_shas[major_version] only once

* Run version specific tests in their own workspace

- Copy version_specific_test_dir to its own folder
- Add WORKSPACE file with scala version to be tested
- Removed tests that don't need to be run for multiple versions

* remove more non version-specific tests

* Remove version from scala labels fix tests

- Add deps required for third_party and tut tests

* Use default_runtime_classpath for macros, fix aspect test

* Run more tests in test_version.sh

* Update naming, default_compile_classpath -> base_compile_classpath

* Use base_classpath instead of base_compile.. + base_runtime...

Using different base classpaths for compile/runtime
requires changes in common.bzl: collect_jars()

Use the same base_classpath for both to limit the scope of
this PR

* Add scala_reflect to default_classpath

* scala_version and shas as on tuple paramter

- Combine two parameters into one tuple to force
  the user the specify shas

* Reduce amount of version specific junit tests

* Run version specific tests

* Run linter

* Remove dynamic creation of scala_default ScalacProvider

- Avoid having to symlink java sources for scalac_worker
- Allows us to only have one scala version at a time

* PR comments, clearer naming

* Use ctx.attr._scala_provider instead of ..attr.scalalib, run linter

* Address comments, clean up WORKSPACE.template

* Remove unused code

* Add section about different scala versions to the readme

README.md

@@ -61,6 +61,25 @@ test --strategy=Scalac=worker
 ```
 to your command line, or to enable by default for building/testing add it to your .bazelrc.
 
+## Selecting Scala version
+
+Rules scala supports all minor versions of Scala 2.11/2.12. By default `Scala 2.11.12` is used and to use another
+version you need to
+specify it when calling `scala_repositories`. `scala_repositories` takes a tuple `(scala_version, scala_version_jar_shas)`
+as a parameter where `scala_version` is the scala version and `scala_version_jar_shas` is a `dict` with
+`sha256` hashes for the maven artifacts `scala_library`, `scala_reflect` and `scala_compiler`:
+```python
+scala_repositories(("2.12.6", {
+    "scala_compiler": "3e892546b72ab547cb77de4d840bcfd05c853e73390fed7370a8f19acb0735a0",
+    "scala_library": "0b3d6fd42958ee98715ba2ec5fe221f4ca1e694d7c981b0ae0cd68e97baf6dce",
+    "scala_reflect": "6ba385b450a6311a15c918cf8688b9af9327c6104f0ecbd35933cfcd3095fe04"
+}))
+```
+If you're using any of the rules `twitter_scrooge`, `tut_repositories`, `scala_proto_repositories`
+or `specs2_junit_repositories` you also need to specify `scala_version` for them. See `./test_version/WORKSPACE.template`
+for an example workspace using another scala version.
+
+
 ## Bazel compatible versions
 
 | bazel | rules_scala gitsha |

WORKSPACE

@@ -24,26 +24,35 @@ load("//specs2:specs2_junit.bzl", "specs2_junit_repositories")
 
 specs2_junit_repositories()
 
-load("//scala:scala_cross_version.bzl", "scala_mvn_artifact")
+load("//scala:scala_cross_version.bzl", "scala_mvn_artifact", "default_scala_major_version")
 
 # test adding a scala jar:
 maven_jar(
     name = "com_twitter__scalding_date",
-    artifact = scala_mvn_artifact("com.twitter:scalding-date:0.17.0"),
+    artifact = scala_mvn_artifact(
+        "com.twitter:scalding-date:0.17.0",
+        default_scala_major_version(),
+    ),
     sha1 = "420fb0c4f737a24b851c4316ee0362095710caa5",
 )
 
 # For testing that we don't include sources jars to the classpath
 maven_jar(
     name = "org_typelevel__cats_core",
-    artifact = scala_mvn_artifact("org.typelevel:cats-core:0.9.0"),
+    artifact = scala_mvn_artifact(
+        "org.typelevel:cats-core:0.9.0",
+        default_scala_major_version(),
+    ),
     sha1 = "b2f8629c6ec834d8b6321288c9fe77823f1e1314",
 )
 
 # test of a plugin
 maven_jar(
     name = "org_psywerx_hairyfotr__linter",
-    artifact = scala_mvn_artifact("org.psywerx.hairyfotr:linter:0.1.13"),
+    artifact = scala_mvn_artifact(
+        "org.psywerx.hairyfotr:linter:0.1.13",
+        default_scala_major_version(),
+    ),
     sha1 = "e5b3e2753d0817b622c32aedcb888bcf39e275b4",
 )
 

scala/BUILD

@@ -1,3 +1,7 @@
+load(
+    "@io_bazel_rules_scala//scala:providers.bzl",
+    _declare_scalac_provider = "declare_scalac_provider",
+)
 load("//scala:scala_toolchain.bzl", "scala_toolchain")
 
 toolchain_type(
@@ -23,3 +27,22 @@ java_import(
     jars = ["@bazel_tools//tools/jdk:TestRunner_deploy.jar"],
     visibility = ["//visibility:public"],
 )
+
+_declare_scalac_provider(
+    name = "scala_default",
+    default_classpath = [
+        "@io_bazel_rules_scala_scala_library",
+        "@io_bazel_rules_scala_scala_reflect",
+    ],
+    default_macro_classpath = [
+        "@io_bazel_rules_scala_scala_library",
+        "@io_bazel_rules_scala_scala_reflect",
+    ],
+    default_repl_classpath = [
+        "@io_bazel_rules_scala_scala_library",
+        "@io_bazel_rules_scala_scala_reflect",
+        "@io_bazel_rules_scala_scala_compiler",
+    ],
+    scalac = "@io_bazel_rules_scala//src/java/io/bazel/rulesscala/scalac",
+    visibility = ["//visibility:public"],
+)

scala/private/rule_impls.bzl

@@ -13,7 +13,10 @@
 # limitations under the License.
 """Rules for supporting the Scala language."""
 load("@io_bazel_rules_scala//scala:scala_toolchain.bzl", "scala_toolchain")
-load("@io_bazel_rules_scala//scala:providers.bzl", "create_scala_provider")
+load(
+    "@io_bazel_rules_scala//scala:providers.bzl",
+    "create_scala_provider",
+    _ScalacProvider = "ScalacProvider")
 load(
     ":common.bzl",
     "add_labels_of_jars_to",
@@ -132,7 +135,7 @@ def compile_scala(ctx, target_label, output, manifest, statsfile, sources,
                   cjars, all_srcjars, transitive_compile_jars, plugins,
                   resource_strip_prefix, resources, resource_jars, labels,
                   in_scalacopts, print_compile_time, expect_java_output,
-                  scalac_jvm_flags):
+                  scalac_jvm_flags, scalac_provider):
   # look for any plugins:
   plugins = _collect_plugin_paths(plugins)
   dependency_analyzer_plugin_jars = []
@@ -223,15 +226,20 @@ StatsfileOutput: {statsfile_output}
   ctx.actions.write(
       output = argfile, content = scalac_args + optional_scalac_args)
 
+  scalac_inputs, _, scalac_input_manifests = ctx.resolve_command(
+      tools = [scalac_provider.scalac])
+
   outs = [output, statsfile]
-  ins = (compiler_classpath_jars.to_list() + all_srcjars.to_list() +
-         list(sources) + plugins_list + dependency_analyzer_plugin_jars +
-         classpath_resources + resources + resource_jars + [manifest, argfile])
+  ins = (
+      compiler_classpath_jars.to_list() + all_srcjars.to_list() + list(sources)
+      + plugins_list + dependency_analyzer_plugin_jars + classpath_resources +
+      resources + resource_jars + [manifest, argfile] + scalac_inputs)
 
   ctx.actions.run(
       inputs = ins,
       outputs = outs,
-      executable = ctx.executable._scalac,
+      executable = scalac_provider.scalac.files_to_run.executable,
+      input_manifests = scalac_input_manifests,
       mnemonic = "Scalac",
       progress_message = "scala %s" % target_label,
       execution_requirements = {"supports-workers": "1"},
@@ -263,7 +271,8 @@ def try_to_compile_java_jar(ctx, scala_output, all_srcjars, java_srcs,
   providers_of_dependencies = collect_java_providers_of(ctx.attr.deps)
   providers_of_dependencies += collect_java_providers_of(
       implicit_junit_deps_needed_for_java_compilation)
-  providers_of_dependencies += collect_java_providers_of([ctx.attr._scalalib])
+  providers_of_dependencies += collect_java_providers_of(
+      ctx.attr._scala_provider[_ScalacProvider].default_classpath)
   scala_sources_java_provider = _interim_java_provider_for_java_compilation(
       scala_output)
   providers_of_dependencies += [scala_sources_java_provider]
@@ -323,13 +332,13 @@ def _compile_or_empty(ctx, manifest, jars, srcjars, buildijar,
     sources = [
         f for f in ctx.files.srcs if f.basename.endswith(_scala_extension)
     ] + java_srcs
-    compile_scala(ctx, ctx.label, ctx.outputs.jar, manifest,
-                  ctx.outputs.statsfile, sources, jars, all_srcjars,
-                  transitive_compile_jars, ctx.attr.plugins,
-                  ctx.attr.resource_strip_prefix, ctx.files.resources,
-                  ctx.files.resource_jars, jars2labels, ctx.attr.scalacopts,
-                  ctx.attr.print_compile_time, ctx.attr.expect_java_output,
-                  ctx.attr.scalac_jvm_flags)
+    compile_scala(
+        ctx, ctx.label, ctx.outputs.jar, manifest, ctx.outputs.statsfile,
+        sources, jars, all_srcjars, transitive_compile_jars, ctx.attr.plugins,
+        ctx.attr.resource_strip_prefix, ctx.files.resources,
+        ctx.files.resource_jars, jars2labels, ctx.attr.scalacopts,
+        ctx.attr.print_compile_time, ctx.attr.expect_java_output,
+        ctx.attr.scalac_jvm_flags, ctx.attr._scala_provider[_ScalacProvider])
 
     # build ijar if needed
     if buildijar:
@@ -474,15 +483,16 @@ def is_dependency_analyzer_off(ctx):
 # Extract very common code out from dependency analysis into single place
 # automatically adds dependency on scala-library and scala-reflect
 # collects jars from deps, runtime jars from runtime_deps, and
-def _collect_jars_from_common_ctx(ctx, extra_deps = [],
+def _collect_jars_from_common_ctx(ctx,
+                                  base_classpath,
+                                  extra_deps = [],
                                   extra_runtime_deps = []):
 
   dependency_analyzer_is_off = is_dependency_analyzer_off(ctx)
 
-  # Get jars from deps
-  auto_deps = [ctx.attr._scalalib, ctx.attr._scalareflect]
-  deps_jars = collect_jars(ctx.attr.deps + auto_deps + extra_deps,
+  deps_jars = collect_jars(ctx.attr.deps + extra_deps + base_classpath,
                            dependency_analyzer_is_off)
+
   (cjars, transitive_rjars, jars2labels,
    transitive_compile_jars) = (deps_jars.compile_jars,
                                deps_jars.transitive_runtime_jars,
@@ -499,13 +509,14 @@ def _collect_jars_from_common_ctx(ctx, extra_deps = [],
       jars2labels = jars2labels,
       transitive_compile_jars = transitive_compile_jars)
 
-def _lib(ctx, non_macro_lib):
+def _lib(ctx, base_classpath, non_macro_lib):
   # Build up information from dependency-like attributes
 
   # This will be used to pick up srcjars from non-scala library
   # targets (like thrift code generation)
   srcjars = collect_srcjars(ctx.attr.deps)
-  jars = _collect_jars_from_common_ctx(ctx)
+
+  jars = _collect_jars_from_common_ctx(ctx, base_classpath)
   (cjars, transitive_rjars) = (jars.compile_jars, jars.transitive_runtime_jars)
 
   write_manifest(ctx)
@@ -551,10 +562,13 @@ def _lib(ctx, non_macro_lib):
     )
 
 def scala_library_impl(ctx):
-  return _lib(ctx, True)
+  scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
+  return _lib(ctx, scalac_provider.default_classpath, True)
 
 def scala_macro_library_impl(ctx):
-  return _lib(ctx, False)  # don't build the ijar for macros
+  scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
+  return _lib(ctx, scalac_provider.default_macro_classpath,
+              False)  # don't build the ijar for macros
 
 # Common code shared by all scala binary implementations.
 def _scala_binary_common(ctx,
@@ -600,7 +614,8 @@ def _scala_binary_common(ctx,
       runfiles = runfiles)
 
 def scala_binary_impl(ctx):
-  jars = _collect_jars_from_common_ctx(ctx)
+  scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
+  jars = _collect_jars_from_common_ctx(ctx, scalac_provider.default_classpath)
   (cjars, transitive_rjars) = (jars.compile_jars, jars.transitive_runtime_jars)
 
   wrapper = _write_java_wrapper(ctx, "", "")
@@ -616,9 +631,12 @@ def scala_binary_impl(ctx):
   return out
 
 def scala_repl_impl(ctx):
+  scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
   # need scala-compiler for MainGenericRunner below
   jars = _collect_jars_from_common_ctx(
-      ctx, extra_runtime_deps = [ctx.attr._scalacompiler])
+      ctx,
+      scalac_provider.default_repl_classpath,
+  )
   (cjars, transitive_rjars) = (jars.compile_jars, jars.transitive_runtime_jars)
 
   args = " ".join(ctx.attr.scalacopts)
@@ -666,8 +684,11 @@ def _scala_test_flags(ctx):
 def scala_test_impl(ctx):
   if len(ctx.attr.suites) != 0:
     print("suites attribute is deprecated. All scalatest test suites are run")
+
+  scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
   jars = _collect_jars_from_common_ctx(
       ctx,
+      scalac_provider.default_classpath,
       extra_runtime_deps = [
           ctx.attr._scalatest_reporter, ctx.attr._scalatest_runner
       ],
@@ -731,8 +752,10 @@ def scala_junit_test_impl(ctx):
     fail(
         "Setting at least one of the attributes ('prefixes','suffixes') is required"
     )
+  scalac_provider = ctx.attr._scala_provider[_ScalacProvider]
   jars = _collect_jars_from_common_ctx(
       ctx,
+      scalac_provider.default_classpath,
       extra_deps = [
           ctx.attr._junit, ctx.attr._hamcrest, ctx.attr.suite_label,
           ctx.attr._bazel_test_runner

scala/providers.bzl

@@ -24,3 +24,35 @@ def create_scala_provider(ijar, class_jar, compile_jars,
       transitive_runtime_jars = transitive_runtime_jars,
       transitive_exports = []  #needed by intellij plugin
   )
+
+ScalacProvider = provider(
+    doc = "ScalaProvider",
+    fields = [
+        "scalac",
+        "default_classpath",
+        "default_macro_classpath",
+        "default_repl_classpath",
+    ])
+
+def _declare_scalac_provider(ctx):
+  return [
+      ScalacProvider(
+          scalac = ctx.attr.scalac,
+          default_classpath = ctx.attr.default_classpath,
+          default_repl_classpath = ctx.attr.default_repl_classpath,
+          default_macro_classpath = ctx.attr.default_macro_classpath,
+      )
+  ]
+
+declare_scalac_provider = rule(
+    implementation = _declare_scalac_provider,
+    attrs = {
+        "scalac": attr.label(
+            executable = True,
+            cfg = "host",
+            allow_files = True,
+            mandatory = True),
+        "default_classpath": attr.label_list(allow_files = True),
+        "default_repl_classpath": attr.label_list(allow_files = True),
+        "default_macro_classpath": attr.label_list(allow_files = True),
+    })