class: center, middle # Generic derivation<br/><span class="title-subtitle">(the hard parts)</span> <h2 class="title-org">Travis Brown<br/><span class="title-org"><a href="https://twitter.com/travisbrown">@travisbrown</a></span> </h2> --- layout: true <div class="footer"> <div class="footer-title"> <h1><span class="sas">Generic derivation: the hard parts</span> | <a href="https://twitter.com/travisbrown">@travisbrown</a></h1> </div> </div> --- .left-column[ ## Intro ] .right-column[ ### Useful links * https://meta.plasm.us/slides/scalaworld/ * https://github.com/travisbrown/circe * https://github.com/travisbrown/circe-derivation ] --- .left-column[ ## Intro ] .right-column[ ### Goal: demystify generic type class derivation "We looked at circe-generic but it seemed too magical." "We were afraid of all the warnings about SI-7046." "We tried it but the compilation times were unacceptable." "We didn't want to butcher our ADTs to fit a JSON schema." ] --- .left-column[ ## Intro ] .right-column[ ### Questions * What is generic derivation? * How do we make it predictable / robust? * How do we make it elegant? * How do we make it fast? * How do we make it configurable? ] --- .left-column[ ## Intro ] .right-column[ ### As well as… * Why are there all these `Lazy`s everywhere? * Wtf does it mean to "export" something? * Why are you using macros to do this thing? * Why does my code take 20 minutes to compile? * How do I get snake-case keys? ] --- .left-column[ ## Intro ] .right-column[ ### Prereqs I'll assume some knowledge of the following: * Scala fundamentals: syntax, implicits, etc. * Algebraic data types (case classes and sealed traits) * Type classes (and their encoding in Scala) * Shapeless's `HList` and `Coproduct` * The existence of the Scala macro system ] --- .left-column[ ## Intro ] .right-column[ ### Prereqs I'll assume some knowledge of the following: * Scala fundamentals: syntax, implicits, etc. * Algebraic data types (case classes and sealed traits) * Type classes (and their encoding in Scala) * Shapeless's `HList` and `Coproduct` * The existence of the Scala macro system Useful resources: * Sam Halliday's [Shapeless for Mortals](https://github.com/fommil/shapeless-for-mortals) * Miles Sabin's [Codecs for Free!](https://www.youtube.com/watch?v=LaOiBdrM8uo) * Lars Hupel's [Deriving Type Class Instances (Part 1)](http://typelevel.org/blog/2013/06/24/deriving-instances-1.html) * My [Type Classes and Generic Derivation](https://meta.plasm.us/posts/2015/11/08/type-classes-and-generic-derivation/) ] --- .left-column[ ## Intro ] .right-column[ ### Our case study Two type classes in [circe](http://circe.io): * `Encoder[A]`: converts a Scala type `A` into JSON * `Decoder[A]`: converts JSON into a Scala type `A` <img src="images/circe.svg" alt="circe" width="60%" align="center" style="margin-top: -2em; padding-left: 2em"/> ] --- .left-column[ ## Intro ] .right-column[ ### Side note #1 `Encoder` and `Decoder`: characterize types of kind `*`: * `Int` * `String` * `Map[Long, List[User]]` ] --- .left-column[ ## Intro ] .right-column[ ### Side note #1 `Encoder` and `Decoder`: characterize types of kind `*`: * `Int` * `String` * `Map[Long, List[User]]` `Functor` and `Monad`: characterize types of kind `* -> *`: * `Option` * `List` * `Task` We'll focus on deriving the former, but both are possible: https://github.com/milessabin/kittens ] --- .left-column[ ## Intro ] .right-column[ ### Following along We'll be looking at several implementations: * Plain macros (no Shapeless) * A simple Shapeless-based implementation * An implementation using Shapeless's type class type class * A more sophisticated Shapeless implementation * The current circe-generic implementation https://github.com/travisbrown/circe-derivation ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### What is generic derivation? Suppose we want to serialize this case class: ```scala case class User(id: Long, handle: String, display: String) ``` Into JSON like this: ```json { "id": 1, "handle": "wordsworth", "display": "WW" } ``` ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### What is generic derivation? Suppose we want to serialize this case class: ```scala case class User(id: Long, handle: String, display: String) ``` Into JSON like this: ```json { "id": 1, "handle": "wordsworth", "display": "WW" } ``` We can write instances by hand… ```scala import io.circe.{ Encoder, Json } implicit val encodeUser: Encoder[User] = Encoder.instance { case User(id, handle, display) => Json.obj( "id" -> Json.fromLong(id), "handle" -> Json.fromString(handle), "display" -> Json.fromString(display) ) } ``` ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### What is generic derivation? This is 100% mechanical boilerplate * We have to write it * We have to maintain it ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### What is generic derivation? This is 100% mechanical boilerplate * We have to write it * We have to maintain it We'd like to avoid this if possible ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### What is generic derivation? Generic derivation lets us just write this: ```scala import io.circe.Encoder, io.circe.generic.auto._ case class User(id: Long, handle: String, display: String) ``` And then: ```scala scala> Encoder[User].apply(User(1, "wordsworth", "WW")) res0: io.circe.Json = { "id" : 1, "handle" : "wordsworth", "display" : "WW" } ``` ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### What is generic derivation? circe-generic is doing the following _at compile time_: * inspecting the structure of the type * finding `Encoder` instances for the members * creating an instance for the case class This also works for: * nested case classes * algebraic data types (ADTs) * anything with a `shapeless.LabelledGeneric` instance ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### Why "generic"? We're abstracting over case classes and sealed families * The standard library doesn't really support this * Shapeless's `Generic` does <div class="tweet"> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><p lang="en" dir="ltr">What is generic programming? <a href="https://twitter.com/milessabin">@milessabin</a>: "kinds of polymorphism that your language doesn't support yet." <a href="https://t.co/tmV23s9zwA">https://t.co/tmV23s9zwA</a></p>— Travis Brown (@travisbrown) <a href="https://twitter.com/travisbrown/status/639863150033158145">September 4, 2015</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> </div> ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### Side note #2 Why not use runtime reflection here? * json4s does it * spray-json does it * Pretty much every Java serialization library does it ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### Side note #2 Why not use runtime reflection here? * Less precision for characterizing what is serializable * Often requires relying on undocumented behavior * Having types supports some advanced features ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### Side note #2 Why not use runtime reflection here? * Less precision for characterizing what is serializable * Often requires relying on undocumented behavior * Having types supports some advanced features We have a rich type system; let's use it if we can ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### So what is generic derivation? Using generic programming to generate type class instances * Map case classes and ADTs to a generic representation * Describe how to create instances for those representations ] --- .left-column[ ## Intro ## What is it? ] .right-column[ ### One constraint before we start Some of my design principles for circe: * The core should have minimal dependencies * Generic derivation should be opt-in * Generic derivation should be pluggable What this means: * No Shapeless in circe-core * No derivation in the `Encoder` / `Decoder` companion objects This will make our lives hard for the next ~35 minutes ] --- .left-column[ ## Intro ## What is it? ## Raw macros ] .right-column[ ### Approach #1: raw macros (no Shapeless) * Everything we'll talk about today relies on macros * You _can_ use the macro API directly * Generally this involves skipping the generic part ] --- .left-column[ ## Intro ## What is it? ## Raw macros ] .right-column[ ### Approach #1: raw macros (no Shapeless) * [Argonaut](https://github.com/argonaut-io/argonaut) * Play JSON's [Inception](https://www.playframework.com/documentation/2.1.1/ScalaJsonInception) * Hanns Holger Rutz's [play-json-sealed](https://github.com/Sciss/play-json-sealed) ] --- .left-column[ ## Intro ## What is it? ## Raw macros ] .right-column[ ### Approach #1: raw macros (no Shapeless) #### Advantages: * No extra dependencies * Compile times are relatively fast ] --- .left-column[ ## Intro ## What is it? ## Raw macros ] .right-column[ ### Approach #1: raw macros (no Shapeless) #### Advantages: * No extra dependencies * Compile times are relatively fast #### Disadvantages: * Macros are experimental, complicate version updates ] --- .left-column[ ## Intro ## What is it? ## Raw macros ] .right-column[ ### Approach #1: raw macros (no Shapeless) #### Advantages: * No extra dependencies * Compile times are relatively fast #### Disadvantages: * Macros are experimental, complicate version updates * It's hard ] --- .left-column[ ## Intro ## What is it? ## Raw macros ] .right-column[ ### Approach #1: raw macros (no Shapeless) #### Advantages: * No extra dependencies * Compile times are relatively fast #### Disadvantages: * Macros are experimental, complicate version updates * It's hard ``` That entry seems to have slain the compiler. Shall I replay your session? I can re-run each line except the last one. [y/n] ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ] .right-column[ ### Approach #1: raw macros (no Shapeless) #### Advantages: * No extra dependencies * Compile times are relatively fast #### Disadvantages: * Macros are experimental, complicate version updates * It's hard ``` That entry seems to have slain the compiler. Shall I replay your session? I can re-run each line except the last one. [y/n] ``` <hr/> #### Conclusion: nope ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ] .right-column[ ### Approach #2: straightforward Shapeless ```scala import io.circe.Decoder import shapeless._, labelled.{ FieldType, field } implicit val decodeHNil: Decoder[HNil] = Decoder.const(HNil) implicit def decodeHCons[K <: Symbol, H, T <: HList](implicit key: Witness.Aux[K], decodeH: Decoder[H], decodeT: Decoder[T] ): Decoder[FieldType[K, H] :: T] = Decoder.instance { c => for { h <- c.get[H](key.value.name) t <- decodeT(c) } yield field[K](h) :: t } implicit def decodeGeneric[A, R](implicit gen: LabelledGeneric.Aux[A, R], decodeR: Decoder[R] ): Decoder[A] = decodeR.map(gen.from) ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ] .right-column[ ### Approach #2: straightforward Shapeless We're inductively describing how to create `Decoder` instances #### Base case For an empty hlist: always succeed with `HNil` #### Inductive case For an `H :: T` where we can decode both parts: * Look up the value at the label for `H` in the JSON object * Decode that JSON value as an `H` * Decode the entire JSON object as a `T` * Cons these pieces into an `H :: T` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ] .right-column[ ### Approach #2: straightforward Shapeless Now we know how to decode hlists! ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ] .right-column[ ### Approach #2: straightforward Shapeless Now we know how to decode hlists! Who cares? ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ] .right-column[ ### Approach #2: straightforward Shapeless Now we know how to decode hlists! Who cares? `HList` is our generic representation of case classes Shapeless's `Generic` gives us our mapping to this representation: ```scala case class User(id: Long, handle: String, display: String) ``` Becomes… ```scala FieldType["id", Long] :: FieldType["handle", String] :: FieldType["display", String] :: HNil ``` (Note that this literal syntax currently requires [Typelevel Scala](https://github.com/typelevel/scala)) ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ] .right-column[ ### Approach #2: straightforward Shapeless This is beautiful—it just works<sup>*</sup>: ```scala scala> case class User(id: Long, handle: String, display: String) defined class User scala> val json = | """{"id": 1, "handle": "wordsworth", "display": "WW"}""" json: String = ... scala> println(io.circe.jawn.decode[User](json)) Right(User(1,wordsworth,WW)) ``` <small><sup>*</sup>…for this example.</small> ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ] .right-column[ ### Approach #2: straightforward Shapeless We can do the same thing for `Encoder` as for `Decoder` ```scala import io.circe.{ Encoder, JsonObject, ObjectEncoder } implicit val encodeHNil: ObjectEncoder[HNil] = ObjectEncoder.instance(_ => JsonObject.empty) implicit def encodeHCons[K <: Symbol, H, T <: HList](implicit key: Witness.Aux[K], encodeH: Encoder[H], encodeT: ObjectEncoder[T] ): ObjectEncoder[FieldType[K, H] :: T] = ObjectEncoder.instance { case h :: t => (key.value.name -> encodeH(h)) +: encodeT.encodeObject(t) } implicit def encodeGeneric[A, R](implicit gen: LabelledGeneric.Aux[A, R], encodeR: ObjectEncoder[R] ): ObjectEncoder[A] = ObjectEncoder.instance(a => encodeR.encodeObject(gen.to(a))) ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ] .right-column[ ### Approach #2: straightforward Shapeless We can also do the same thing for ADTs as for case classes All four kinds of instances look pretty much the same (See [HListCodecs.scala](https://github.com/travisbrown/circe-derivation/blob/master/core/src/main/scala/io/circe/examples/derivation/simple/HListCodecs.scala) and [CoproductCodecs.scala](https://github.com/travisbrown/circe-derivation/blob/master/core/src/main/scala/io/circe/examples/derivation/simple/CoproductCodecs.scala) for details) ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ] .right-column[ ### Approach #3: a beautiful dead end #### Observation: * Our `Encoder` and `Decoder` derivations look similar * So do implementations for `Eq`, `Show`, `Arbitrary`, etc. * These are all type classes ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ] .right-column[ ### Approach #3: a beautiful dead end #### Observation: * Our `Encoder` and `Decoder` derivations look similar * So do implementations for `Eq`, `Show`, `Arbitrary`, etc. * These are all type classes #### Conclusion: * Clearly we need a type class type class ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ] .right-column[ ### Approach #3: a beautiful dead end ```scala val typeClass: LabelledTypeClass[Encoder] = new LabelledTypeClass[Encoder] { val emptyProduct: Encoder[HNil] = ... def product[H, T <: HList]( name: String, encodeH: Encoder[H], encodeT: Encoder[T] ): Encoder[H :: T] = ... val emptyCoproduct: Encoder[CNil] = ... def coproduct[L, R <: Coproduct]( name: String, encodeL: => Encoder[L], encodeR: => Encoder[R] ): Encoder[L :+: R] = ... def project[F, G]( instance: => Encoder[G], to: F => G, from: G => F ): Encoder[F] = ... } ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ] .right-column[ ### Approach #3: a beautiful dead end This seems fantastic—18 tweets worth of code gives us this: ```scala scala> import io.circe.syntax._ ... scala> import io.circe.examples.derivation.dryer.encoders._ ... scala> case class Foo(a: Int, b: Option[String]) ... scala> val json = Foo(1234, Some("abcdef")).asJson.noSpaces json: String = {"a":1234,"b":{"Some":{"x":"abcdef"}}} scala> import io.circe.examples.derivation.dryer.decoders._ ... scala> println(io.circe.jawn.decode[Foo](json)) Right(Foo(1234,Some(abcdef))) ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ] .right-column[ ### Reality Our `LabelledTypeClass` approach is great, except… * Instances for multiple type classes don't play well together * Derived instances will smash explicitly-defined ones * It doesn't work in some cases because of definition order * Sometimes it just doesn't work and you don't know why ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ] .right-column[ ### Our type class type class and colliding instances ```scala scala> import io.circe.examples.derivation.dryer.encoders._ import io.circe.examples.derivation.dryer.encoders._ scala> import io.circe.examples.derivation.dryer.decoders._ import io.circe.examples.derivation.dryer.decoders._ scala> case class Foo(a: Int, b: Option[String]) defined class Foo scala> import io.circe.syntax._ import io.circe.syntax._ scala> val json = Foo(1234, Some("abcdef")).asJson.noSpaces <console>:22: error: could not find implicit value for parameter encoder: io.circe.Encoder[Foo] val json = Foo(1234, Some("abcdef")).asJson.noSpaces ^ ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ] .right-column[ ### Our type class type class and colliding instances Our choices if we want to use `LabelledTypeClass`: * Just don't support deriving both at once * Put the instances in our companion objects * Copy-paste a lot of the machinery Shapeless provides ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ] .right-column[ ### Our type class type class and colliding instances Our choices if we want to use `LabelledTypeClass`: * Just don't support deriving both at once * Put the instances in our companion objects * Copy-paste a lot of the machinery Shapeless provides <hr/> None of these are viable options We've DRY-ed ourselves into a corner #### Conclusion: nope ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ] .right-column[ ### More reality Our more straightforward approach before is great, except… * It doesn't work for recursive data types * Derived instances still smash explicitly-defined ones * It also doesn't work in some cases because of definition order * Sometimes it also just doesn't work and you don't know why ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ] .right-column[ ### Recursive data types Given a simple recursive ADT: ```scala import io.circe.examples.derivation.simple._ sealed trait IntList case object IntNil extends IntList case class IntCons(h: Int, t: IntList) extends IntList ``` Derivation will fail: ```scala scala> io.circe.Decoder[IntList] <console>:19: error: diverging implicit expansion for type io.circe.Decoder[IntList] starting with method decodeMapLike in object Decoder io.circe.Decoder[IntList] ^ ``` This would work just fine for e.g. a non-recursive `IntOption` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ] .right-column[ ### Recursive data types The solution: Shapeless's `Lazy`: ```scala implicit def decodeGeneric[A, R](implicit gen: LabelledGeneric.Aux[A, R], decodeR: Lazy[Decoder[R]] ): Decoder[A] = decodeR.value.map(gen.from) ``` This is why you see Shapeless's `Lazy` all over circe-generic (See [Shapeless for Mortals](https://github.com/fommil/shapeless-for-mortals) for more discussion) ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ] .right-column[ ### Recursive data types Okay, that one was easy ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### Derived instances meet explicitly-defined instances ```scala scala> import io.circe.syntax._ import io.circe.syntax._ scala> List(1, 2, 3).asJson.noSpaces res0: String = [1,2,3] scala> import io.circe.examples.derivation.simple._ import io.circe.examples.derivation.simple._ scala> List(1, 2, 3).asJson.noSpaces res1: String = {"::":{"head":1,"tl$1":[2,3]}} ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### Derived instances meet explicitly-defined instances Scala's implicit resolution looks in two places: * identifiers in the current scope (including imports) * the companion objects of "associated" types The former always win, but… ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### Derived instances meet explicitly-defined instances Scala's implicit resolution looks in two places: * identifiers in the current scope (including imports) * the companion objects of "associated" types The former always win, but… * we have to import `encodeGeneric` * `encodeGeneric` provides instances for case classes * `scala.List` is a case class * circe's instance for `List` is in the `Encoder` companion object ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### Derived instances meet explicitly-defined instances Scala's implicit resolution looks in two places: * identifiers in the current scope (including imports) * the companion objects of "associated" types The former always win, but… * we have to import `encodeGeneric` * `encodeGeneric` provides instances for case classes * `scala.List` is a case class * circe's instance for `List` is in the `Encoder` companion object …so exactly the wrong thing happens ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### Derived instances meet explicitly-defined instances This is generally a problem with "orphan instances" in Scala ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### The spray-json / [spray-json-shapeless](https://github.com/fommil/spray-json-shapeless) solution Orphan _all_ instances * Instances for e.g. `List` live in a "protocol" trait * Derived instances live in another trait * User-defined instances live in yet another trait (or object) * Prioritization is worked out through inheritance ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### The spray-json / [spray-json-shapeless](https://github.com/fommil/spray-json-shapeless) solution Orphan _all_ instances * Instances for e.g. `List` live in a "protocol" trait * Derived instances live in another trait * User-defined instances live in yet another trait (or object) * Prioritization is worked out through inheritance <hr/> I don't like imports I want everything possible in the codec companion objects #### Conclusion: nope ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### The [argonaut-shapeless](https://github.com/alexarchambault/argonaut-shapeless) solution Shapeless has this evidence thingy called `LowPriority`: ```scala import io.circe.{ Encoder, Json } import shapeless.LowPriority implicit def fallbackEncoder[A](implicit lowPriority: LowPriority ): Encoder[A] = Encoder.instance(_ => Json.obj()) ``` This will encode anything with no existing `Encoder` This is in many ways a nice, simple, general solution, but… ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### The [argonaut-shapeless](https://github.com/alexarchambault/argonaut-shapeless) solution Shapeless has this evidence thingy called `LowPriority`: ```scala import io.circe.{ Encoder, Json } import shapeless.LowPriority implicit def fallbackEncoder[A](implicit lowPriority: LowPriority ): Encoder[A] = Encoder.instance(_ => Json.obj()) ``` This will encode anything with no existing `Encoder` This is in many ways a nice, simple, general solution, but… <hr/> I like my type classes to have type parameters #### Conclusion: nope ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### The [export-hook](https://github.com/milessabin/export-hook) solution From circe: ```scala object Decoder extends LowPriorityDecoders { ... } trait LowPriorityDecoders { implicit def importedDecoder[A](implicit exported: Exported[Decoder[A]] ): Decoder[A] = exported.instance } ``` Now derivers can sneak low-priority instances into scope * Any derivation provider can export instances * They'll always be appropriately prioritized ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### A moment of reflection So far we've recapitulated the history of circe up to 0.4.0 * We've solved a divergence problem with `Lazy` * We've solved a priority problem with exports * Now [everything works great](https://github.com/travisbrown/circe-derivation/tree/master/core/src/main/scala/io/circe/examples/derivation/better) ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ] .right-column[ ### A moment of reflection So far we've recapitulated the history of circe up to 0.4.0 * We've solved a divergence problem with `Lazy` * We've solved a priority problem with exports * Now [everything works great](https://github.com/travisbrown/circe-derivation/tree/master/core/src/main/scala/io/circe/examples/derivation/better) …except that compilation has gotten really, really slow ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Compilation speed Exercise: * Fire up a console in a project with a circe 0.3.0 dependency * Copy and paste the [quick start example](https://github.com/travisbrown/circe/tree/v0.3.0#quick-start) from the README * Count out how long it takes the last two lines to compile ```scala import io.circe._, io.circe.generic.auto._ import io.circe.parser._, io.circe.syntax._ sealed trait Foo case class Bar(xs: List[String]) extends Foo case class Qux(i: Int, d: Option[Double]) extends Foo val foo: Foo = Qux(13, Some(14.0)) foo.asJson.noSpaces decode[Foo](foo.asJson.spaces4) ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Side note: runtime vs. compile-time Runtime performance of derived instances: * Some overhead due to intermediate generic representations * Generally not enough to worry about (see [benchmarks](https://github.com/travisbrown/circe/pull/96)) ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Improving compilation speed Two approaches: * Help users avoid deriving things more than once * Make the actual process of derivation faster ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Where derivation happens How many times does this code derive `Encoder[Foo]`? ```scala import io.circe.examples.derivation.better.auto._ import io.circe.syntax._ case class Foo(a: Int, b: Option[String]) val json1 = Foo(1, Some("abc")).asJson val json2 = Foo(2, Some("def")).asJson val json3 = Foo(3, None).asJson ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Where derivation happens How many times does this code derive `Encoder[Foo]`? ```scala import io.circe.examples.derivation.better.auto._ import io.circe.syntax._ case class Foo(a: Int, b: Option[String]) val json1 = Foo(1, Some("abc")).asJson val json2 = Foo(2, Some("def")).asJson val json3 = Foo(3, None).asJson ``` Answer: 2 too many ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Where derivation happens What about here? ```scala import io.circe.examples.derivation.better.auto._ import io.circe.syntax._ case class Foo(a: Int, b: Option[String]) case class Bar(foo1: Foo, foo2: Foo, foo3: Foo) io.circe.Encoder[Bar] ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Semi-automatic derivation circe-generic contains two packages: * `auto`: import contents and derivation just happens * `semiauto`: provides derivers you have to invoke ```scala import io.circe.generic.semiauto._, io.circe.syntax._ case class Foo(a: Int, b: Option[String]) object Foo { implicit val encodeFoo = deriveEncoder[Foo] } val json1 = Foo(1, Some("abc")).asJson val json2 = Foo(2, Some("def")).asJson val json3 = Foo(3, None).asJson ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Semi-automatic derivation * Less magical than `auto` * Only a little extra boilerplate * Can drastically reduce compile times ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Making derivation happen faster Remember how our derivation works: * We describe how to build hlist / coproduct instances * Implicit search works in reverse to find instances ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Making derivation happen faster ```scala import io.circe.generic.derivation.better._ case class Foo(a: Int) val instance = io.circe.Decoder[Foo] ``` * Implicit search for `Foo` instance hits `decodeGeneric` * …triggering a `FieldType["a", Int] :: HNil` instance search * …which hits `decodeHCons` * …triggering searches for `Int` and `HNil` instances * The `Int` search finds `Decode.decodeInt` * The `HNil` search finds `decodeHNil` * …and we're done ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Making derivation happen faster This all relies on general purpose implicit search A slightly faster approach (for now): * Use a macro materializer for generic representations * Fold over the hlist or coproduct directly * Don't repeat searches for repeated types ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Making derivation happen faster This all relies on general purpose implicit search A slightly faster approach (for now): * Use a macro materializer for generic representations * Fold over the hlist or coproduct directly * Don't repeat searches for repeated types Soon the compiler may do this itself (thanks, [Miles](https://github.com/typelevel/scala#roadmap)) ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Making derivation happen faster This approach was [introduced in circe 0.4.0](https://github.com/travisbrown/circe/pull/247) * Test suite compile times cut in half * Reports of even larger improvements from some users * Small runtime performance gains (+5% throughput) ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Making derivation happen faster This approach was [introduced in circe 0.4.0](https://github.com/travisbrown/circe/pull/247) * Test suite compile times cut in half * Reports of even larger improvements from some users * Small runtime performance gains (+5% throughput) …but not everything is perfect—macros are still hard * [Hygiene issues](https://github.com/travisbrown/circe/pull/299) * [Sneaky exponential growth](https://github.com/travisbrown/circe/pull/366) ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Case class decoder derivation in circe 0.5.1 <img src="images/old-cc-small.png" width="100%" align="center" /> ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Case class decoder derivation in circe 0.5.1 <img src="images/old-cc.png" width="100%" align="center" /> ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Case class decoder derivation in circe 0.5.2 <img src="images/new-cc.png" width="100%" align="center" /> ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Case class decoder derivation in circe 0.5.2 (depth) <img src="images/new-depth.png" width="100%" align="center" /> ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### ADT decoder derivation in circe 0.5.2 <img src="images/new-adt.png" width="100%" align="center" /> ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ] .right-column[ ### Making derivation happen faster This will still be painful in some situations Some room for improvement (at both compiler and library level): * Smarter search * Better caching ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ] .right-column[ ### Most requested circe features * "I want my field names turned into snake-case keys" * "I want to use a type field as a discriminator" * "I want case objects serialized as strings" * "Why doesn't my decoder respect default values?" ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ] .right-column[ ### Possible approaches to configurability * Use "hinting" type classes (e.g. [spray-json-shapeless](https://github.com/fommil/spray-json-shapeless/blob/master/src/main/scala/fommil/sjs/FamilyFormats.scala)) * Make derivers configurable * Make everything type-driven ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ] .right-column[ ### Hinting type classes * `ProductHint` / `CoproductHint` type classes represent options * Generic instances provide defaults * Users can override defaults with custom instances ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ] .right-column[ ### Hinting type classes * `ProductHint` / `CoproductHint` type classes represent options * Generic instances provide defaults * Users can override defaults with custom instances <hr/> This approach is totally reasonable It's probably the "right" approach for Scala I don't like the idea of using implicit scope for configuration ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ] .right-column[ ### Configurable derivers * Derivation mechanisms are meant to be pluggable in circe * Users configure a deriver and import its contents * The `auto` package object is just the default deriver ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ] .right-column[ ### Configurable derivers * Derivation mechanisms are meant to be pluggable in circe * Users configure a deriver and import its contents * The `auto` package object is just the default deriver <hr/> #### Advantages: * It's easy compared to the next option #### Disadvantages: * It's probably not sufficiently flexible for real-world use ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ] .right-column[ ### Type-driven configuration * Our codec type classes have an extra configuration type * A deriver can choose to provide and honor config types * Users can request configuration in their constraints * Users can configure instances with `semiauto` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ] .right-column[ ### Type-driven configuration ```scala import io.circe._, io.circe.jawn._, io.circe.syntax._ import io.circe.generic.auto._, io.circe.generic.config._ case class Foo(thisIsSomeNumber: Int = 0) ``` And then: ```scala scala> Foo(1).asJsonConfigured[SnakeCaseKeys].noSpaces res0: String = {"this_is_some_number":1} scala> decodeConfigured[Foo, SnakeCaseKeys](res0) res1: cats.data.Xor[io.circe.Error,Foo] = Right(Foo(1)) scala> decodeConfigured[Foo, UseDefaultValues]("{}") res2: cats.data.Xor[io.circe.Error,Foo] = Right(Foo(0)) ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ] .right-column[ ### Implementation note These configuration types are _essentially_ phantom types They are instantiated and they do work, but this isn't user-facing ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ] .right-column[ ### Why it's taking so long This approach introduces lots of issues: * Should config cascade in the case of generic types? (yes) * Should explicitly-defined instances still always win? (yes) And lots of corner cases: * e.g. a configurable type inside a generic type with an explicitly-defined instance inside a configurable type: ```scala case class Foo(someField: Int) case class Foos(foos: List[Foo]) ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ## Conclusion ] .right-column[ ### The future of generic derivation in circe * Configurability (very soon) * Caching for `auto` instances * Compiler improvements (SI-7046, inductive implicits) ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ## Conclusion ] .right-column[ ### Thanks! * https://github.com/travisbrown/circe * https://github.com/travisbrown/circe-derivation * https://gitter.im/travisbrown/circe ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ## Conclusion ## Appendix ] .right-column[ ### Appendix: deriving incomplete decoders Given: ```scala case class User(id: Long, name: String, email: String) ``` And incoming JSON from a `POST` request: ```json { "name": "Foo McBar", "email": "foo@mcbar.com" } ``` Boilerplate-y solution: ```scala case class PartialUser(name: String, email: String) { def toUser(id: Long): User = User(id, name, email) } ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ## Conclusion ## Appendix ] .right-column[ ### Appendix: deriving incomplete decoders Since circe 0.2: ```scala import io.circe.generic.auto._, io.circe.jawn._ case class User(id: Long, name: String, email: String) val posted = """{ "name": "Foo McBar", "email": "foo@mcbar.com" }""" val func = decode[Long => User](posted) val createdUser = func.map(_(101L)) ``` And then: ```scala scala> createdUser.foreach(println) User(101,Foo McBar,foo@mcbar.com) ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ## Conclusion ## Appendix ] .right-column[ ### Appendix: deriving patching decoders Incoming JSON from a `PATCH` request: ```json { "email": "foo@mcbar.com" } ``` Boilerplate-y solution: ```scala case class PatchedUser( name: Option[String], email: Option[String] ) ``` ] --- .left-column[ ## Intro ## What is it? ## Raw macros ## Shapeless! ## Trying to DRY ## Problems ## Priorities ## Speed ## Configuration ## Conclusion ## Appendix ] .right-column[ ### Appendix: deriving patching decoders In circe: ```scala import io.circe.generic.auto._, io.circe.jawn._ case class User(id: Long, name: String, email: String) val posted = """{ "email": "foo@example.com" }""" val user = User(101, "Foo McBar", "foo@mcbar.com") val func = decode[User => User](posted) val patchedUser = func.map(_(user)) ``` And then: ```scala scala> patchedUser.foreach(println) User(101,Foo McBar,foo@example.com) ``` ]