Scala at Scale at Twitter

class: center, middle

# Scala at Scale at Twitter

<h2 class="title-title"><a href="https://twitter.com/travisbrown">@travisbrown</a><br/><span class="title-org">Twitter, Inc.</span>
</h2>
<h2 class="title-venue"><span>OSCON</span> 2015</h2>

---

layout: true

<div class="footer">
  <div class="footer-title">
    <h1><span class="sas">Scala at Scale at Twitter</span> | <a href="https://twitter.com/travisbrown">@travisbrown</a></h1>
  </div>
</div>

---

.left-column[
  ## Intro
]
.right-column[
### My background

* Working primarily in Scala since 2010
* Active in projects like [Shapeless](https://github.com/milessabin/shapeless) and [Scalaz](https://github.com/scalaz/scalaz)
* 890 answers in the [Scala tag](http://stackoverflow.com/questions/tagged/scala) on Stack Overflow
* Joined the Open Source team at Twitter in May 2014

### What I do at Twitter

* Represent external adopters of our OSS libraries
* Work on Scala projects (with a focus on [Finagle](https://twitter.github.io/finagle/))
* Teach [Twitter University](https://twitter.com/university) classes
* Organize events (like [FinagleCon](http://finagle.github.io/finaglecon/)!)
]

---

.left-column[
  ## Intro
]
.right-column[
### Not really the focus of this talk:

* Not primarily a history of Scala at Twitter
* Not primarily about runtime performance of Scala

### Instead:

* The development of a Scala _culture_ at scale
* Our relationship to the larger Scala ecosystem
]

---

.left-column[
  ## Intro
  ## Scala?
]
.right-column[
### A multi-paradigm language for the JVM

* Object oriented

```scala
class Calculator(brand: String) {
  def add(m: Int, n: Int): Int = m + n
}

class ScientificCalculator(brand: String)
  extends Calculator(brand) {
    def log(m: Double, base: Double) =
      math.log(m) / math.log(base)
}
```

* Functional

```scala
Stream.from(1).filterNot(_ % 2 == 0).map { i =>
  i * i
}
```
]

---

.left-column[
  ## Intro
  ## Scala?
]
.right-column[
### A few of Scala's language features

* Java interoperability
* Mixin composition
* Singleton objects
* Generics (with variance)
* Higher-order functions
* Higher-kinded types
* Type classes
* Implicit values and conversions
* Monadic for-comprehensions
* Pattern matching
* Structural types
* Existential types
* Path-dependent types
* Macros
]

---

.left-column[
  ## Intro
  ## Scala?
]
.right-column[
### The “better Java” end of the spectrum

Use case classes, pattern matching, and a few functional idioms, but avoid
advanced type system features, implicits, too much functional stuff, etc.

### The “worse Haskell” end

Avoid subtyping, embrace abstraction, try to push everything you can into the
type system.

```scala
import scalaz._, Scalaz._

def powerSet[A](xs: List[A]) = xs.filterM { _ =>
  true :: false :: Nil
}
```
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
]
.right-column[

### Alex Payne [in 2010](http://www.slideshare.net/al3x/the-how-and-why-of-scala-at-twitter) on early use of Scala at Twitter
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
]
.right-column[

### Before Scala: the Monorail in 2009

Monolithic Ruby on Rails application proved hard to scale
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
]
.right-column[

* January 2010: First email to [scala-cafe]
* August 2010: Introduction of `Future` and `Try`
* May 2011: First [Scalding](https://github.com/twitter/scalding) commit
* December 2011: Introduction of ScalaTest
* February 2012: Publication of [Effective Scala](http://twitter.github.io/effectivescala/)
* May 2012: Migration to Scala 2.9
* September 2012: Accelerated “off the Monorail” effort
* December 2012: Introduction of ScalaCheck
* September 2014: Migration to Scala 2.10

<blockquote class="twitter-tweet" lang="en" width="550"><p lang="en" dir="ltr">I've decided to transition into a role heading up special projects.</p>— The Monorail (@monorail_repo) <a href="https://twitter.com/monorail_repo/status/614186349810221056">June 25, 2015</a></blockquote>
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
]
.right-column[

### More history

* [Getting Off the Monorail or: How Twitter Learned to Stop Worrying and Love Monads](https://vimeo.com/53693402) (2012) by Jeff Sarnat
* [Scala at Twitter](https://www.youtube.com/watch?v=4GlvhOccd1A) (2015) by Stu Hood and Brian Degenhardt
* [From Source: Scala at Twitter](https://www.parleys.com/tutorial/from-source-scala-twitter) (2015) by Stu Hood
* [Scala: The Industrial Parts](http://monkey.org/~marius/scala2015.pdf) (2015) by Marius Eriksen
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
]
.right-column[

### The codebase

* Several million lines of Scala (35% of codebase)
* Internal monorepo and monobuild (as of Monday)

### Internal / external

* About 40 Scala projects on GitHub
* A few are GitHub-only; consumed as third-party
* Most are kept in sync with directories in monorepo
* Built with [Pants](https://pantsbuild.github.io/) (internal) and [SBT](http://www.scala-sbt.org/) (open source)

### Third-party dependencies

* [ScalaTest](http://scalatest.org/), [ScalaCheck](https://www.scalacheck.org/), and [Scalastyle](http://www.scalastyle.org/)
* Not many runtime Scala dependencies
* Still using our own `Future` and `Try`
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
]
.right-column[

### Scala is a big, complex language

* Lots of ways to do _everything_
* Powerful abstractions (at a cost)

### The language and ecosystem change rapidly

* Scala 2.9 → Scala 2.10

### Not all of our developers have deep experience in Scala

* Can't assume background in FP
* Many teams work primarily in Java (25% of codebase)
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
]
.right-column[

### Consistency and the monobuild

> “With a monorepo, you just refactor the API and all of its callers in one commit. That’s not always trivial, but it’s much easier than it would be with lots of small repos. I’ve seen APIs with thousands of usages across hundreds of projects get refactored and with a monorepo setup it’s so easy that no one even thinks twice.”

<div class="attrib">— Dan Luu, <a href="http://danluu.com/monorepo/">Advantages of Monolithic Version Control</a></div>
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
]
.right-column[

### Consistency and the monobuild

The following are all exactly equivalent in Scala:

```scala
val inc1 = (i: Int) => i + 1
val inc2 = (_: Int) + 1
val inc3: Int => Int = _ + 1
val inc4: Int ⇒ Int = _ + 1
val inc5 = new Function1[Int, Int] {
  def apply(i: Int) = i + 1
}
```

### Impact on cross-team collaboration

* Cognitive burden
* Lack of confidence in refactoring
* Lots of space for bikeshedding

And it's not just syntax: inconsistency at the level of abstractions and paradigms
can be as bad or worse

]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
]
.right-column[

### The “wrong” answer

* Throw all advanced abstractions out the window
* Go with the lowest common denominator everywhere

### Fundamental principles and abstractions

* Prefer immutability
* Avoid implicit conversions
* Save exceptions for exceptional circumstances
* Use futures for concurrency
* Model services as `A => Future[B]`

### We want to build a culture around these principles
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
]
.right-column[

### Documentation

* [Scala School](https://twitter.github.io/scala_school/) and [Effective Scala](http://twitter.github.io/effectivescala/)
* Scala Guide (currently internal-only)
* Tech Radar by Twitter Architecture Group

### Process and tooling

* Code review, internal Q & A, chat
* Scalastyle, Scalariform, linting options

### Training

* Twitter University created in 2013 
* We offer dozens of Scala courses
  * Scala Essentials (12 hours)
  * Advanced Scala series (10 courses)
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
]
.right-column[

### Advanced Scala training

* Functional programming in Scala
* Scala collections
* Scala-Java interoperability
* Monads, functors, and applicative functors in Scala
* Scala reflection
* Scala macros
* Advanced Scala type system
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
]
.right-column[

### Scala-Java interoperability

* Java from Scala: almost perfect
* Scala from Java: currently very tricky

### Tools and conventions

* Immutability-oriented [collections converters](https://github.com/twitter/util/blob/master/util-core/src/main/java/com/twitter/util/javainterop/Scala.java)
* `AbstractX` class wrappers for Scala traits
* Java “companion objects”: `FuturePool` / `FuturePools`
* Java compilation tests

`.MODULE$.` “only” appears ~750 times in our Java code

]

---
.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
]
.right-column[

### Sharp edges

* Variable shadowing (both values and types)
* Unclear type-checking errors
* Baffling syntactic sugar
* Initialization order

<blockquote class="twitter-tweet" lang="en" width="550"><p lang="en" dir="ltr">Another day, another question on the internal <a href="https://twitter.com/hashtag/scala?src=hash">#scala</a> mailing list where the answer is "You've just met value discarding. We're so sorry."</p>— Travis Brown (@travisbrown) <a href="https://twitter.com/travisbrown/status/574974356473843712">March 9, 2015</a></blockquote>
]

---
.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
]
.right-column[

### Not a perfect solution (but on the whole it works)

<blockquote class="twitter-tweet" lang="en" width="550"><p lang="en" dir="ltr">Getting started with macros now that we're on 2.10 cc <a href="https://twitter.com/jco">@jco</a> <a href="http://t.co/QXcyGrjWCU">pic.twitter.com/QXcyGrjWCU</a></p>— (ﾉ`□´)ﾉ⌒ʇləʌəıN əoſ (@joenievelt) <a href="https://twitter.com/joenievelt/status/507988268244152320">September 5, 2014</a></blockquote>
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
  ## Outside world
]
.right-column[

### Twitter's Scala projects outside of Twitter

<div class="logos">
  <div><img src="images/finagle.png" width="28%"/></div>
  <div style="padding-right: 1.35em;"><img src="images/finatra.png" width="25%"/></div>
  <div><img src="images/summingbird_logo.png" width="25%"/></div>
</div>
<div class="logos">
  <div><img src="images/scalding.png" style="padding-left: 2.5em; padding-right: 1.35em;" width="40%"/></div>
  <div><img src="images/zipkin.jpg" width="25%"/></div>
</div>
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
  ## Outside world
]
.right-column[

### [Finagle adopters](https://github.com/twitter/finagle/blob/master/ADOPTERS.md)

* Box
* EA DICE
* Foursquare
* Nest
* The National Center for Biotechnology Information
* The New York Times
* PagerDuty
* Pinterest
* Rdio
* Sabre Labs
* SoundCloud
* SproutSocial
* StumbleUpon
* Strava
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
  ## Outside world
]
.right-column[

### Impediments to external adoption

* Slow upgrades (one year to full support for 2.11)
* Many of our projects [don't do semantic versioning](https://finagle.github.io/blog/2014/10/07/finagle-releases/)
* SBT isn't primary internal build system
* Parallel types: we have our own `Future` and `Try`
]

---
.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
  ## Outside world
]
.right-column[

### Two futures

This is more “Not Invented Yet” than NIH: our future implementation preceded [SIP-14](http://docs.scala-lang.org/sips/completed/futures-promises.html) by over a year.

### Why we still use our own

* Fewer context switches
* Interruptibility
* Support for [continuation-local variables](https://github.com/ReactiveX/RxJava/issues/2885)
* (Something like) tail-call elimination

### Current status

* We provide and support [converters](https://github.com/twitter/bijection/tree/master/bijection-util/src/main/scala/com/twitter/bijection/twitter_util)
* We will adopt the standard library's `Try`
* We _may_ implement `concurrent.Future` at some point

]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
  ## Outside world
]
.right-column[

### Impediments to open source use at Twitter

> Scalaz is great… [but] we carefully benchmark and tune (wrt GC and CPU) many of these libraries (especially Util, Finagle, Algebird), and regressions in those areas would be unacceptable to runtime systems. Guaranteeing this is more of a challenge with libraries that we don't maintain actively.

<div class="attrib">— Oscar Boykin, <a href="http://www.reddit.com/r/IAmA/comments/23s80n/we_work_on_open_source_at_twitter_ask_us_anything">Twitter OSS Reddit AMA</a></div>

]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
  ## Outside world
  ## The future
]
.right-column[

### New experiments in co-ownership

* [The Finagle organization](https://github.com/finagle/) (community hub)
* [algebra](https://github.com/non/algebra) (unifying [Algebird](http://github.com/twitter/algebird) and [Spire](http://github.com/non/spire))
* [OpenZipkin](https://github.com/openzipkin/zipkin) (community-driven fork)

### [Finch](https://github.com/finagle/finch): Finagle meets non-Twitter Scala libraries

* Depends on Shapeless and [Argonaut](http://argonaut.io/)
* May depend on [cats](https://github.com/non/cats) (a Scalaz alternative)
* 6+ adopters using Finch in production
* 29 contributors (25 external)
]

---

.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
  ## Outside world
  ## The future
]
.right-column[

### Things we're excited about

* Faster Scala version upgrades (~month, not year+)
* [Abide](https://github.com/scala/scala-abide): better linting for Scala
* [scala.meta](http://scalameta.org/) for large-scale refactoring tools
* Clearer upgrade paths via [MiMa](https://github.com/typesafehub/migration-manager)
* Java 8 optimizations and interop [in Scala 2.12](https://github.com/scala/scala-java8-compat)
* SBT 1.0 for more maintainable OSS builds
]

---
.left-column[
  ## Intro
  ## Scala?
  ## Why Scala?
  ## History
  ## Challenges
  ## Culture
  ## Outside world
  ## The future
  ## We're hiring!
]
.right-column[

## Thanks!

<blockquote class="twitter-tweet" lang="en" width="550"><p lang="en" dir="ltr">Want to build libraries that hundreds of millions of people rely on every day? Here's your chance: <a href="https://t.co/BOV3Pdm6f8">https://t.co/BOV3Pdm6f8</a> /cc <a href="https://twitter.com/JoinTheFlock">@JoinTheFlock</a></p>— Finagle (@finagle) <a href="https://twitter.com/finagle/status/616696300734578688">July 2, 2015</a></blockquote>
]