Tooling: resolve all project dependencies
Jul 7, 2019
2 minute read

To provide good tooling support we can’t just rely on all source files of a project. There are hardly any projects which come along without any additional dependencies. Understanding these dependencies is the key to better understand the user code when building static analysis tools.

Fortunately the jvm ecosystem is so large that we can leverage existing tools to build a facade which will provide us all transitive dependencies we need to understand a project better.

For the sake of this blog post we assume that the project to analyze is a Gradle project. We now that the Gradle build system can provide us all user declared dependencies:

 /g/r/detekt/detekt-cli{} $ gradle -q dependencies --configuration implementation                                                                     1s 375ms

------------------------------------------------------------
Project :detekt-cli
------------------------------------------------------------

implementation - Implementation only dependencies for compilation 'main' (target  (jvm)). (n)
+--- org.jetbrains.kotlin:kotlin-stdlib:1.3.40 (n)
+--- project detekt-core (n)
+--- com.beust:jcommander:1.74 (n)
\--- org.jetbrains.kotlin:kotlin-compiler-embeddable:1.3.40 (n)

(n) - Not resolved (configuration is not meant to be resolved)

We parse this output to get maven like coordinates for the dependencies:

val artifacts = content.splitToSequence("\n")
    .filter { "\\--- " in it || "+--- " in it }
    .map { it.substringAfter("---") }
    .map { it.trim() }
    .map { it.split(" ") }
    .mapNotNull { if (it.isNotEmpty()) it[0] else null }
    .filter { it.split(":").size == 3 }
    .toSet()

artifacts.forEach(::println)
// org.jetbrains.kotlin:kotlin-stdlib:1.3.40
// com.beust:jcommander:1.74
// org.jetbrains.kotlin:kotlin-compiler-embeddable:1.3.40

These are the declared dependencies, however all transitive dependencies are still missing. We need to resolve them in a second step. For this I wrote a small maven-resolver wrapper library: deps. It accepts maven like coordinates and looks up maven local ($HOME/.m2/repository) or does a roundtrip to maven central or bintray.

val artifacts = listOf(
    "org.jetbrains.kotlin:kotlin-stdlib:1.3.40",
    "com.beust:jcommander:1.74",
    "org.jetbrains.kotlin:kotlin-compiler-embeddable:1.3.40"
).map { Artifact(it) }

val paths = Deps().resolve(artifacts)

paths.forEach(::println)
// /home/abosch/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib/1.3.40/kotlin-stdlib-1.3.40.jar
// /home/abosch/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib-common/1.3.40/kotlin-stdlib-common-1.3.40.jar
// /home/abosch/.m2/repository/org/jetbrains/annotations/13.0/annotations-13.0.jar
// /home/abosch/.m2/repository/com/beust/jcommander/1.74/jcommander-1.74.jar
// /home/abosch/.m2/repository/org/jetbrains/kotlin/kotlin-compiler-embeddable/1.3.40/kotlin-compiler-embeddable-1.3.40.jar
// /home/abosch/.m2/repository/org/jetbrains/kotlin/kotlin-script-runtime/1.3.40/kotlin-script-runtime-1.3.40.jar
// /home/abosch/.m2/repository/org/jetbrains/kotlin/kotlin-reflect/1.3.40/kotlin-reflect-1.3.40.jar
// /home/abosch/.m2/repository/org/jetbrains/intellij/deps/trove4j/1.0.20181211/trove4j-1.0.20181211.jar

Here now we have a full list of all dependencies which are needed to build detekt-cli.

From here on we can convert this list to a classpath string and pass it to e.g. the Kotlin compiler:

val classpath = paths.joinToString(":")

call("kotlinc -classpath $classpath")

Here now ends our roundtrip to resolve all transitive dependencies for a Gradle project.