I recently ran some workshops at work around getting to the heart of functional programming:
- pure/total functions
- referential transparency
- local reasoning
- type driven development
- effects
Much of the inspiration came from Rob Norris' excellent talk on Functional programming with effects. This is not the talk I presented at work; however, I did put together an addendum to that talk that walked through some of the same effects highlighted as Rob but run through the same parametric function that was generic on the effect. I did not look very hard but I have not seen anything else quite like it so here it goes for a blog post. This is also a blog about monadic and applicative properties.
SETUP
The example we will use is a generic function that takes in three of the same
context / effect / container (use whatever term works best in your brain),
combines them in order to create a User
, and then returns that User
wrapped
in the same context / effect / whatever. The F[_]: Monad
means give me any
context / effect / whatever that implements Monad (flatMap, bind, whatever).
case class User(name: String, id: Long, age: Double) object Thing { def doThing[F[_]: Monad](a: F[String], b: F[Long], c: F[Double]): F[User] = for { aa <- a bb <- b cc <- c } yield User(aa, bb, cc) }
We will run doThing
over and over by supplying values wrapped in different
effects: Option, Either, Future, etc. Remember that the above for comprehension
is not a for loop and desugars into:
def doThing[F[_]: Monad](a: F[String], b: F[Long], c: F[Double]): F[User] = a.flatMap(aa => b.flatMap(bb => c.map(cc => User(aa, bb, cc)))) }
Also remember that Monad is just:
trait Monad[F[_]] { // some way to put a value into a monad def pure[A](value: A): F[A] // some way to collapse the nested monad down // e.g. I am mapping some fn over my context but that fn also returns the context def flatMap[A, B](value: F[A])(func: A => F[B]): F[B] }
Effects
Option
Intuition:
- handling partial functions where other languages may throw
- gives us back total functions when something can go wrong
// option object ValidOptionThing { val oName: Option[String] = Some("mat") val oId: Option[Long] = Some(17828382L) val oAge: Option[Double] = Some(1.3) } object InvalidOptionThing { val oName: Option[String] = None val oId: Option[Long] = ValidOptionThing.oId val oAge: Option[Double] = ValidOptionThing.oAge } // Some(User(mat, 1782382, 1.3)) object validOThing extends App { val res = Thing.doThingS(ValidOptionThing.oName, ValidOptionThing.oId, ValidOptionThing.oAge) println(s"res: $res") } // None object invalidOThing extends App { val res = Thing.doThing(InvalidOptionThing.oName, InvalidOptionThing.oId, InvalidOptionThing.oAge) println("we short circuit since aa is none, b and c never evaluated") println(s"res: $res")
- When we run this with three filled options we get back
Some(User..)
- When we run this with one option that is None, we short-circuit and get
None
The compiler turns the generic doThing
function into something like the
following at compile time:
def doThing(a: Option[String], b: Option[Long], c: Option[Double]): Option[User] = for { aa <- a bb <- b cc <- c } yield User(aa, bb, cc)
I will skip pointing out this translation step in the remainder of the examples.
Either
Intuition:
- handling partial functions where other languages may throw
- gives us back total functions when something can go wrong
- gives us back exceptions where we can say what went wrong
object ValidEitherThing { type VEThing[A] = Either[Throwable, A] val oName: VEThing[String] = "mat".asRight val oId: VEThing[Long] = 17828382L.asRight val oAge: VEThing[Double] = 1.3.asRight } object InvalidEitherThing { type VEThing[A] = Either[Throwable, A] val oName: VEThing[String] = new java.lang.NoSuchFieldError("nope").asLeft val oId: VEThing[Long] = 17828382L.asRight val oAge: VEThing[Double] = 1.3.asRight } // Right(User(mat,17828382,1.3)) object validEThing extends App { val res = Thing.doThing(ValidEitherThing.oName, ValidEitherThing.oId, ValidEitherThing.oAge) println(s"res: $res") } // Left(NoSuchFieldError("nope")) object invalidEThing extends App { val res = Thing.doThing(InvalidEitherThing.oName, InvalidEitherThing.oId, InvalidEitherThing.oAge) println("we short circuit since aa is none, b and c never evaluated") println(s"res: $res") }
- When we have three
Right
values we get backRight(User..)
- When we have one or more
Left
values we short-circuit and getLeft(what went wrong)
Future
Intuition:
- something that is happening concurrently, possibly on another thread, like a network call
- also has a failure channel
Note: Future is not referentially transparent, prefer to use IO
or Task
// pretend these are actually long running results running elsewhere object ValidFutureThing { val name: Future[String] = Future.successful("mat") val id: Future[Long] = Future.successful(17828382L) val age: Future[Double] = Future.successful(1.3) } object InvalidFutureThing { object applicativeInvalidTThing extends App { val name: Future[String] = Future.failed(new TimeoutException("nope")) val id: Future[Long] = Future.successful(17828382L) val age: Future[Double] = Future.successful(1.3) } object validFThing extends App { import ExecutionContext.Implicits.global val res = Thing.doThing(ValidFutureThing.name, ValidFutureThing.id, ValidFutureThing.age) val r = Await.result(res, Duration.Inf) println(s"res: $r") } object invalidFThing extends App { import ExecutionContext.Implicits.global val res = Thing.doThing(InvalidFutureThing.name, InvalidFutureThing.id, InvalidFutureThing.age) val r = Await.result(res, Duration.Inf) println(s"res: $r")
- When we run this with 3 successul futures, we get
Future[User(..)]
- When we run this with one failed future, say one of those futures like
aa
times out we short circuit sinceaa
failed
Task
Intuition: The same as Future
// Task object ValidTaskThing { val name: Task[String] = Task.now("mat") val id: Task[Long] = Task.now(17828382L) val age: Task[Double] = Task.now(1.3) } object InvalidTaskThing { val name: Task[String] = Task.raiseError(new TimeoutException("nope")) val id: Task[Long] = Task.now(17828382L) val age: Task[Double] = Task.now(1.3) } object validTThing extends App { val res = Thing.doThing(ValidTaskThing.name, ValidTaskThing.id, ValidTaskThing.age) val task = res.runAsync val r = Await.result(task, Duration.Inf) println(s"res: $r") } object invalidTThing extends App { val res = Thing.doThing(InvalidTaskThing.name, InvalidTaskThing.id, InvalidTaskThing.age) val task = res.runAsync val r = Await.result(task, Duration.Inf) println(s"res: $r") }
- we get the same result as the Future case
- note since
a
times out,b
andc
are never evaluated
Aside - Applicative
In the above examples, a
, b
, c
don't depend on each other however, we have
sequenced them due to flatMap. Since they have nothing to do with each other,
we can use applicative rather than monadic behavior here. E.g. we want to
run a sequence of independent computations and combine the result. Sadly, in
Scala, no more For
syntax; however, we can write something very similar to the
original doThing
method replacing the Monad
constraint with an Applicative
constraint:
object ApplicativeThing { def doThingA[F[_]: Applicative](a: F[String], b: F[Long], c: F[Double]): F[User] = (a, b, c).mapN { case (aa, bb, cc) => User(aa, bb, cc) }
If we run something like the failing task example, we get much the same answer as before:
object ApplicativeThing { object applicativeInvalidTThing extends App { val res = ApplicativeThing.doThingA(InvalidTaskThing.name, InvalidTaskThing.id, InvalidTaskThing.age) val task = res.runAsync val r = Await.result(task, Duration.Inf) println(s"res: $r")
We still blow up with a timed out task; The big difference here is that
unlike the monadic case where b
and c
are never evaluated, b
and c
do
get evaluted here. It's just the combination that fails at the end.
Aside - Applicative Validation
If we have independent effects, we can combine them like an Either but accumulate everything that went wrong rather than just the first thing that went wrong.
object ValidValidationThing { type NelThing[A] = ValidatedNel[String, A] val oName: NelThing[String] = "mat".validNel val oId: NelThing[Long] = 17828382L.validNel val oAge: NelThing[Double] = 1.3.validNel } object InvalidValidationThing { type NelThing[A] = ValidatedNel[String, A] val oName: NelThing[String] = "username invalid".invalidNel val oId: NelThing[Long] = 17828382L.validNel val oAge: NelThing[Double] = "age invalid too".invalidNel }
If you try to use this monadically it is a compiler error, since ValidationNel
has no Monad.
object validVThing extends App { val res = Thing.doThing(ValidValidationThing.oName, ValidValidationThing.oId, ValidValidationThing.oAge) println(s"res: $res") } // :( compiler error)
But ValidationNel
does have an Applicative
:
object validVThing extends App { val res = ApplicativeThing.doThingA(ValidValidationThing.oName, ValidValidationThing.oId, ValidValidationThing.oAge) println(s"res: $res") } object invalidVThing extends App { val res = ApplicativeThing.doThingA(InvalidValidationThing.oName, InvalidValidationThing.oId, InvalidValidationThing.oAge) println("we short circuit since aa is none, b and c never evaluated") println(s"res: $res") }
- When we have 3 valid elements, we get
Valid(User..)
- When we have 2 of three elements as invalid we get:
Invalid(NonEmptyList(username invalid, age invalid too))
- we all all the errors with no short-circuiting
- if we had used Either we would only see the username error
Aside - Nested Task with Either
What if we end up with a Task[Either[Throwable, User]]
?
- thing.doThing
doesn't work! it's a compiler error
We would have to write something like this with two for comprehensions:
object nestedTaskEither { type NestedTask[A] = Task[Either[Throwable, A]] def doThingNested( a: NestedTask[String], b: NestedTask[Long], c: NestedTask[Double]): NestedTask[User] = for { aa <- a bb <- b cc <- c } yield for { aaa <- aa bbb <- bb ccc <- cc } yield User(aaa, bbb, ccc)
This sucks but it does work:
object Nested extends App { import nestedTaskEither._ val teName: NestedTask[String] = Task.now("mat".asRight) val teId: NestedTask[Long] = Task.now(17828382L.asRight) val teAge: NestedTask[Double] = Task.now(1.3.asRight) val res = doThingNested(teName, teId, teAge) val task = res.runAsync val r = Await.result(task, Duration.Inf) println(s"res: $r")
How do we make this work with the original DoThing
without this
annoying double unpacking of an effect inside an effect? We can use a monad
transformer like EitherT to
get us back to where we want to be. EitherT
knows about the task inbetween:
object monadTransformer extends App { type TE[A] = EitherT[Task, Throwable, A] val mtName: TE[String] = EitherT(Task.now("mat".asRight)) val mtId: TE[Long] = EitherT(Task.now(17828382L.asRight)) val mtAge: TE[Double] = EitherT(Task.now(1.3.asRight)) // call our original method at the top of the file that only // has the single for comprehension val res = Thing.doThing(mtName, mtId, mtAge) val task = res.value.runAsync val r = Await.result(task, Duration.Inf) println(s"res: $r") }
aaaaaaaaaaaaaaaaaaand it all works again and we get back Right(User(...))
Back to Monads and Effects
The previous examples were all about failure. Which is cool. This is super useful for us. But there is way more to Monads than that, including an entire book
List
This one doesn't really make that much sense with our running example. List imbues the effect of multiple possible results. We get all the possible results with List.
object listThing extends App { val userNames = List("mat", "steve", "jim") val ids = List(1L) val ages = List(1.3, 2.7, 99.9) val res = Thing.doThing(userNames, ids, ages) println(s"res: $res") }
In the previous example we just got one result. In this case, we get all the combinations of our inputs:
User(mat,1,1.3)
User(mat,1,2.7)
User(mat,1,99.9)
User(steve,1,1.3)
User(steve,1,2.7)
User(steve,1,99.9)
User(jim,1,1.3)
User(jim,1,2.7)
User(jim,1,99.9)
Reader
Intuition: dependency injection
It's true that we can just use constructors for dependency injection in scala. This is often the right idea, but not the only way to do dependency injection. See pros and cons here and here
Reader (and it's Monad) let us sequence operations that depend on some input:
object readerThing extends App { // some config or whatever class we are injecting case class Injected(version: String, idShift: Long, ageShift: Double) // two different versions of the thing we want to inject val injected = Injected("3.2", 200000L, 37.2) val someOtherInjected = Injected("9.9", 382973238L, 99.9) // the inputs to doThing type Config[A] = Reader[Injected, A] val rName: Config[String] = Reader(in => s"${in.version}:mat") val rId: Config[Long] = Reader(in => 17828382L + in.idShift) val rAge: Config[Double] = Reader(in => in.ageShift + 1.3) // this is the "program" returned from running doThing // it hasn't done anything yet, it's really more like // a function that is waiting for an input before the result // e.g. program is a function of Injected => User val program = Thing.doThing(rName, rId, rAge) // this just gives us the composed "program" waiting for an input // we need to supply the input val runRes = progam.run(injected) println(s"res: $runRes") // run it with some other config injected val runRes2 = program.run(someOtherInjected) println(s"res: $runRes2")
- now this is super interesting
- runRes uses injected and give sback
User(3.2:mat, 180....)
- runRes2 injects something else entirely and gives back
User(9.9:mat, ...)
- runRes uses injected and give sback
- notice how making the program
Thing.doThing(rName, rId, rAge)
didn't do anything- it just guves us back
Reader[Injected, User]
but not the actual user
- it just guves us back
- we need to supply it with the config we want for it to work
- we can re-use the same program by applying different configs and getting different users
Writer
Intuition: we want to run some computation and while we are doing that we want to annoate that computation.
- feels like tracing and logging (but doesn't have to be, and don't actually use it for logging)
object writerThing extends App { case class Computation(notes: String, money: Int) type Trace[A] = Writer[List[Computation], A] val wtName: Trace[String] = for { a <- "mat".pure[Trace] _ <- List(Computation("fetched user", 100)).tell } yield a val wtId: Trace[Long] = for { a <- 17827382L.pure[Trace] _ <- List(Computation("fetched id", 1000)).tell } yield a val wtAge: Trace[Double] = for { a <- 1.3.pure[Trace] _ <- List(Computation("fetched age", 10000)).tell } yield a val program = Thing.doThing(wtName, wtId, wtAge) val (notes, user) = program.run println(s"trace: $notes \n ----------\n") println(s"user: $user") }
this results in:
notes: List(Computation(fetched user,100), Computation(fetched id,1000), Computation(fetched age,10000))
----------
user: User(mat,17827382,1.3)
- running this through our generic
doThing
will create a User just like the other examples - it also annotates it with the extra computation information
Conclusions
Phheww. We looked at a bunch of effects. Some of those effects short-circuited, some of those effects did some work somewhere else (possibly on another thread), and some of those effects were very different from the rest:
- we ran some computation in F resulting in an F[User]
- the effect is whatever differentiates User from F[User]
We can keep going with these building blocks. For example, in the Reader example we would keep composing more readers and only at the end of the world (edge of the program) do we supply the initial config value. This is very cool. This is a very different way to think about programs and putting together programs in a referentially transparent way.