Generating Twirp clients using Square's Wire and Retrofit
In this post we'll explore how we can make use of square's wire library & gradle plugin to generate some retrofit interfaces that are capable of making calls to twirp services from an android app. If you'd like to see a simple sample project using all the pieces that we'll discuss in this post you can find it here
An RPC by any other name...
What's twirp you ask? Don't worry I asked that same question a few weeks ago as I began to explore how to do everything we're about to go over in this post.
Let's go right to the source for explaining what Twirp is
Twirp is a simple RPC framework built on protobuf. You define a service in a .proto specification file, then Twirp will generate servers and clients for that service. It's your job to fill in the "business logic" that powers the server, and then generated clients can consume your service straight away.
Ok cool, so it's just another way of making some network calls from our android app to our back end services. Similar to gRPC.
If we go to the twirp github page we'll see that they list some libraries and tools that can help you get started in a variety of languages. Sadly for us in the android world, Kotlin is not on the list, and Java only has two entries. Worse still, only one of the Java entries is capable of generating client side implemenations, which is what we need for an android app. That sole implementation makes use of the protoc compiler and jaxrs to handle the client side implementation of a twirp service.
That's unfortunate because as android developers we'd rather not generate some really gnarly java objects with protoc and combine them with a jaxrs client implementation inside of an android app. That sounds like a lot of bloat and a lot of headache.
OK well before we go that not so android-y route, let's see if we can use some of the tools and libraries we're all familiar with in the android community.
If we were using regular gRPC we know we'd reach for square's wire library at this point.
If we wanted to just make http POST calls with json objects in the body, we'd reach for retrofit
With that in mind, let's go see what's possible with wire, retrofit, and twirp.
That's so meta(programming)
If we take a look at the main page for the wire project we'll see pretty quickly that they have included a way to customize the output of the code they generate from protoc files. That's pretty cool, because it's exactly what we want to do.
They provide a way for you to create a custom "schema handler" that can output generated code based on the protoc files that you feed into wire.
In a perfect scenario, we would let wire generate all of the data/message objects, and we would just step in to generate the service objects, rather than letting wire generate its usual gRPC compliant service objects.
Let's see if we can make this dream a reality...
We'll need to create a SchemaHandler class and luckily for us the wire project includes a few "recipes" for schema handlers. These examples can be a great jumping off point for making your own.
Our schema handler is going to ignore all protoc defintions except for service defintions. We're only interested in making sure we can generate a retrofit call for every RPC call in our collection or protoc files.
This mean's we only need to implement one of the three handle methods defined in the SchemaHandler class.
So without further ado, let's look at what a schema handler that can generate retrofit interfaces looks like, and then we'll explain the meta programming madness going on :)
class TwirpRetrofitServiceSchemaHandler : SchemaHandler() {
override fun handle(type: Type, context: SchemaHandler.Context): Path? {
return null
}
override fun handle(service: Service, context: SchemaHandler.Context): List<Path> {
return listOf(writeRetrofitInterface(service, context))
}
override fun handle(extend: Extend, field: Field, context: SchemaHandler.Context): Path? {
return null
}
private fun writeRetrofitInterface(service: Service,
context: SchemaHandler.Context) : Path{
val protoType = service.type
val path = context.outDirectory / toKotlinSourceFile(protoType).joinToString(separator = "/")
val packageDotPath = toPackage(protoType)
val methodDeclarations = mutableListOf<String>()
service.rpcs.forEach { rpc ->
methodDeclarations.add(toRetrofitMethod(rpc, packageDotPath, protoType.simpleName))
}
val retrofitInterface = """
package $packageDotPath
import retrofit2.http.Body
import retrofit2.http.Headers
import retrofit2.http.POST
import com.google.protobuf.*
interface ${protoType.simpleName} {
${methodDeclarations.joinToString("\n")}
}
""".trimIndent()
context.fileSystem.createDirectories(path.parent!!)
context.fileSystem.write(path) {
writeUtf8(retrofitInterface)
}
return path
}
private fun toRetrofitMethod(rpc: Rpc,
packagePath: String,
serviceType: String): String{
val endpointPath = "twirp/" + packagePath + "." + serviceType + "/" + rpc.name
val methodName = rpc.name
val requestParameter = rpc.requestType?.simpleName?.lowercase()
val responseType = if(rpc.responseType.toString() == "google.protobuf.Empty"){
"com.google.protobuf.Empty"
} else {
rpc.responseType.toString()
}
val requestType = if(rpc.requestType.toString() == "google.protobuf.Empty"){
"com.google.protobuf.Empty"
} else {
rpc.requestType.toString()
}
return """
@POST("$endpointPath")
@Headers("Content-Type: application/json")
suspend fun $methodName(@Body $requestParameter : $requestType): $responseType
"""
}
private fun toPackage(protoType: ProtoType): String {
val result = mutableListOf<String>()
for (part in protoType.toString().split(".")) {
result += part
}
return result.subList(0, result.size-1).joinToString(".")
}
private fun toKotlinSourceFile(protoType: ProtoType): List<String> {
val result = mutableListOf<String>()
for (part in protoType.toString().split(".")) {
result += part
}
result[result.size - 1] = (result[result.size - 1]) + ".kt"
return result
}
}
The first thing to note, is that we only implement the handle method that gets passed service types. It wants us to return a list path objects. These are the file paths to the retrofit interface source files that we generate.
Let's skip over the meat of the object real fast and talk about the toPackage and toKotlinSourceFile methods. The toPackage method will just take our prototype object and figure out what its kotlin package declaration should be.
The toKoltinSourceFile will do a similar thing, except its going to determine where our retrofit interface will live in the generated folder structure.
Next let's look at our writeRetrofitInterface, this is the heart of our schema handler. It's going to look at the ProtoType object that it gets handed, and convert it into a retrofit interface with a method call for each rpc defined in the protoc defintion.
The first thing it does is take the list of rpc calls and pass each one to the toRetrofitMethod. This method is what writes the retrofit interface method that will make a twirp call at runtime.
Under the hood twirp is just http POST with json request bodies. So all of our retrofit interface methods will be annoted with @POST and specify a content type of application/json. The rest of the method is just taking bits of the RPC object and stuffing them into a string that will one day become a method for this twirp service interface.
** One small thing to note about my implementation here is the special handling of the com.google.Empty object. I needed to make that small tweak because of how the empty object ends up being generated in the package structure after wire is finished. You may not need to do that in your own implementations.
Back in the writeRetrofitInterface we combine the strings from our toRetrofitMethod calls for each RPC into a full retrofit interface declaration. At the end of the method we write to disk, and viola we're done. We've created a retrofit interface from our protoc files and each method in each interface, represents one twirp RPC.
** One more note about my implementation. I am importing all of the com.google.protobuf package to make my life easier in the project I'm working in. If you don't use any of google's protobuf package protoc files, then you can safely remove that import, and in fact you'll need to or you'll get a compile error
Easy right? Nothing to it...
Factories, jars, and gradle configs oh my...
The next step in our schema handler adventure is to define a factory that will return an instance of our schema handler. This is simply required for wire to know how to work with it.
//Who doesn't love a factory class
class TwirpRetrofitServiceSchemaHandlerFactory: SchemaHandler.Factory{
override fun create(): SchemaHandler {
return TwirpRetrofitServiceSchemaHandler()
}
}
We then take these two fine classes, and compile them into a jar that we will add as a buildscript classpath dependency in our android app's root level build.gradle file
dependencies {
//Yes, that is what I named my jar file
classpath files("libs/twirp-retrofit-service-schema-handler.jar")
}
I chose to make a folder call libs at the root of my android project to hold the jar. You could place it where ever your heart desires, as long as you tell gradle where to find it.
Once we've done that we just need to go configure our wire plugin, inside of our app module.
So let's go open the build.gradle file for our app module. In that file we'll add a wire configuration block that looks like this:
wire {
custom {
schemaHandlerFactoryClass = "dev.bltucker.twirp.TwirpRetrofitServiceSchemaHandlerFactory"
exclusive = false
}
kotlin {
android = true
rpcRole = 'none'
}
}
Let's walk through everything that is going on in this config block.
First, we define the custom block. This where we configure wire to use our schema handler factory, which it will then use to create an instance of our schema handler. We set exclusive to false, because we don't want wire to finish after our schema handler is done. We want it to continue on to the kotlin block handler. This is because our schema handler is going to handle protoc declarations specifiying rpc calls. We'll let the regular kotlin process handle message objects in our protoc files.
The next block is the kotlin block. This is where we tell wire how we want it to generate our message objects. We have just two configuration options to set in this block. First, we tell wire we're an android project. This will signal to wire that it should generate nice looking android-y message objects with Parcelable implementations.
Finally, we'll define our RPC role as "none" because we dont need the regularly generated RPC client objects that wire would normally provide.
The full instructions for setting up the wire gradle plugin can be found here and the instructions specific to setting up a custom schema handler, can be found here
I'd highly recommend reading through their documentation when setting this up in your own projects.
Now that we've done all of that, we can hit the big green play button in android studio or issue a build command to gradle via the command line (if you're into that kind of thing), and our android project will build itself and generate all the service and message objects we need to communicate with our twirp backends.
Mixing camel_case & snakeCase
Alright! We're in business now. We've got our custom schema handler plugged into wire. It handles creating the retrofit interfaces that we need to talk to the twirp services defined in our proto files, and the regular wire plugin handles turning all of our Message objects into nice-for-android kotlin objects complete with Parcelable implementations.
Welllll, not so fast. It's pretty common for a proto file to define a field on an object using snake_case. That's not usually a problem though because if you use protoc to generate some java or kotlin objects, it'll be smart enough to convert those snake cased protoc declarations into camelCased fields.
Unfortunantely, the authors of wire decided they wouldnt do that.
In their own words from wire's github readme
Wire avoids case mapping. A field declared as picture_urls in a schema yields a Java field picture_urls and not the conventional pictureUrls camel case. Though the name feels awkward at first, it's fantastic whenever you use grep or more sophisticated search tools. No more mapping when navigating between schema, Java source code, and data. It also provides a gentle reminder to calling code that proto messages are a bit special.
Alright well that's cool, they have a good argument for why they did that, but now they've caused us a little problem. Our json converter is gonna be looking for fields who's name exactly matches the json property. Our snake cased fields won't match any camel cased json properties :(
Normally, we could resolve this by just annotating our fields like this:
@SerializedName("camelCasedJsonProperty")
val silly_snake_cased_field
Sadly, we aren't the one's authoring our objects. Wire is. So what do we do?
Well luckily for us wire slaps a WireField annotation on every field of every object it generates and that annotation includes a jsonName attribute when the field is snake cased and won't match the json.
This means we can create a FieldNamingStrategy for Gson to teach it how to map these fields to and from json.
We'll simply look for the WireField annotation. When we find it, we'll look for the jsonName attribute that we can use to perform the mapping. If we don't find the annotation, or it lacks a jsonName attribute, then we'll just fall back to the regular strategy of assuming the json and field names will match each other.
class WireAnnotatedFieldNamingStrategy : FieldNamingStrategy {
override fun translateName(f: Field): String {
if(!f.isAnnotationPresent(WireField::class.java)){
return f.name
}
val wireFieldAnnotation = f.getAnnotation(WireField::class.java)
return wireFieldAnnotation?.jsonName?.ifEmpty { f.name } ?: f.name
}
}
"Instant" failure
One last thing to do. We might need to make calls that include the wire library's Instant data type as a field. I know my proto files included a few message objects with this type.
Out of the box Gson won't know how to handle doing that. Luckily Gson make's it easy to teach it how to handle serialization and deserialization of a type it doesn't recognize via its TypeAdapter facilities. So we'll make a type adapter for Instant real quick and then we'll be in business!
class InstantTypeConverter : JsonSerializer<Instant>, JsonDeserializer<Instant> {
override fun serialize(src: Instant?, typeOfSrc: Type?, context: JsonSerializationContext?): JsonElement? {
if(src == null){
return null
}
return JsonPrimitive(src.toString());
}
override fun deserialize(json: JsonElement?, typeOfT: Type?, context: JsonDeserializationContext?): Instant? {
if(json == null){
return null
}
return Instant.parse(json.asString)
}
}
And that's it. Your code generation plugs right into your regular android build chain using the wire gradle plugin. No tricks, just a plain android gradle build(ok I dunno if there's anything plain about an android gradle build, but you know what I mean)
You should now be able to make twirp calls from your android app using the retrofit interfaces we all know and love.
That's it for this post. I hope you enjoyed diving into some code generation with me. As always I've included links that I found helpful below