Statically-typed JSON payload in Swift
Increasing signal-to-noise ratio via types
Introduction
Quite a lot has already been said about creating properly typed API clients on top of URLSession. In fact, two of my favorite blog posts about the topic are:
- Writing a Scalable API Client in Swift 5 by my good friend Víctor Pimentel.
- Modern Networking in Swift 5 with URLSession, Combine and Codable by Vadim Bulavin, whose blog has really high-quality content about Swift and iOS.
Unlike Android, where Retrofit seems like the standard way to define API clients (establishing very specific ways to handle the structure of them), most iOS developers end up creating our own abstraction on top of URLSession. Even if Combine provides a better API on top of URLSession (fixing the wrong (Data?, URLSession?, Error?) tuple…), we still have to figure out how to model requests, headers, request bodies, query params, etc…
Every Friday, the iOS team at Jobandtalent gathers together in a weekly meeting, where we usually discuss different topics about the project or iOS/Swift related in general. This time, one of the topics was analyzing the current implementation of our APIClient abstraction (which was created almost five years ago and has aged quite well, to be honest) and how we could improve it.
Then, we opened a playground.
This blog post (which doesn’t intend by any means to be a full-featured API client implementation in Swift) describes the next two hours after we opened that playground…
The problem
It all started by taking a look at our current model for HTTPRequest.
protocol HTTPRequest {
var method: HTTPMethod { get }
var path: String { get }
var headers: [String: String]? { get }
var parameters: [String: Any]? { get }
}enum HTTPMethod: String {
case GET, POST, PUT, DELETE
}
Some of the problems were:
- The parameters were reused both for GET and POST requests. For GET requests, parameters were encoded as query params, whereas for POST requests, they were encoded as a JSON payload inside the HTTP request body. Parameters for GET and POST requests have slightly different requirements, so it doesn’t really make sense to share those.
- The parameters’ type
[String: Any]allowed putting any type as value, even if it doesn’t make sense and cannot be transformed properly to a JSON payload (in case of a POST request), leading to runtime issues. - Nitpick detail, but the ergonomics of using a protocol to model our requests is slightly worse than using a struct and have the request values created via
static letmethods or properties. That way, we could leverage the addicted dot syntax (.|), whenever aHTTPRequestis needed, to list all the available requests. Even if Swift allows creating computed static vars or functions by extending a protocol, you’ll bump into the following error when using themStatic member ‘xxx’ cannot be used on protocol metatype ‘HTTPRequest.Protocol’.
The solution
We went step by step, trying to solve the first and third problems first.
A request could be a simple value, encapsulating the path, method, and the way the data is parsed.
struct Request<Output> {
var path: String
var method: Method
var decode: (Data) -> Output
}Most of the times we want the Data to be parsed to a Decodable model…
extension Request where Output: Decodable {
init(path: String, method: Method = .get()) {
self.init(path: path, method: method) { data in
try! JSONDecoder().decode(Output.self, from: data)
}
}
}… but sometimes, it can be useful to skip altogether the information sent back from the server (via the no-op closure).
extension Request where Output == Void {
init(path: String, method: Method) {
self.init(path: path, method: method) { _ in () }
}
}As stated before, both GET and POST request parameters have different invariants, and the way to maintain those invariants was to properly define different type definitions that made sense for each case.
enum Method {
case get(_ queryItems: [String: CustomStringConvertible] = [:])
case post(_ payload: [String: Any]) var name: String {
switch self {
case .get: return "GET"
case .post: return "POST"
}
} var body: Data? {
switch self {
case .get(queryItems: _):
return nil case .post(payload: let payload):
return try! JSONSerialization.data(
withJSONObject: payload
)
}
}
}
For GET requests, [String: CustomStringConvertible] seemed like a proper, simple type for abstracting query parameters that will later be transformed to a URLQueryItem type.
We could easily declare a GET request by extending the Request type and have static methods or properties inside.
extension Request {
static var allJobs: Request<[Job]> {
.init(path: "jobs")
}
}When trying to use the APIClient to send a request, we could just type a dot whenever a Request type is needed (apiClient.send(request: .|)), having Xcode show the proper completions, which makes it a joy to use and improves the ergonomics of the API over the previously HTTPRequest protocol-based solution.
But as you can see, we still had the second problem we mentioned before. The payload type for POST requests was[String: Any], which could make the JSONSerialization.data(withJSONObject: payload) code crash if we were not careful enough about the types we put there. We thought we could leverage the Swift type system to do better, and have the compiler help us provide correct data that would avoid runtime issues.
We won’t go into much detail about the actual
APIClientclass. It just wrapped a base URL and had a method that acceptsRequest<Output>types, transform them intoURLRequest, and notified via a callback(Request<Output>, (Result<Output, Error>) -> Void) -> Void.
The Encodable type
Using the Encodableand JSONEncoder types seemed like the most idiomatic way to transform types into JSON Data.
enum Method {
case get(_ queryItems: [String: CustomStringConvertible] = [:])
case post(_ payload: Encodable) var body: Data? {
switch self {
case .get(queryItems: _):
return nil case .post(payload: let payload):
return try! JSONEncoder().encode(payload)
}
}
}
Unfortunately, this code didn’t compile, and for very good reasons. The line JSONEncoder().encode(payload) resulted in the following error.
Value of protocol type ‘Encodable’ cannot conform to ‘Encodable’; only struct/enum/class types can conform to protocols.
The problem was that, in order for the compiler to know how to properly convert a type to Data, it needs to know the exact nominal type, as the implementation of func encode(to encoder: Encoder) throws varies depending on the specific type. And how could we let the compiler know about the specific type? Of course, generics…
JSON payload as a generic Encodable type
The introduction of the generic spread all around, making us modify all the API client, from the send(request:) method, up to the Request type.
struct Request<Output, Payload> where Payload: Encodable {
var path: String
var method: Method<Payload>
var decode: (Data) -> Output
}enum Method<Payload> where Payload: Encodable {
case get(_ queryItems: [String: CustomStringConvertible] = [:])
case post(_ payload: Payload) var body: Data? {
switch self {
case .get(queryItems: _):
return nil case .post(payload: let payload):
return try! JSONEncoder().encode(payload)
}
}
}
An interesting option to avoid the impact of adding this new generic all around could have been to use type erasure and create a
AnyEncodabletype.
Once we had all the infrastructure in place, we could create requests like this:
extension Job {
struct Payload {
var title: String
var description: String
var id: Int
}
}extension Request {
static var allJobs: Request<[Job], ???> {
.init(path: "jobs")
}
static func modifyJob(
payload: Job.Payload
) -> Request<Void, Job.Payload> {
.init(
path: "jobs",
method: .post(payload)
)
}
}
As you can see, the first problem that we came across was the Payload type for GET requests… The first thing that might come to mind is to put Void there, but…
type ‘Void’ does not conform to protocol ‘Encodable’
Unfortunately, we couldn’t extend Void to conform to Encodable, as Void is a non-nominal type (it’s an empty tuple in fact).
So, if Void is not an option… was there any other type that could make sense and satisfy the type system for GET requests, where the Payload type doesn’t really make much sense? Luckily for us, yes.
Never is what’s called an uninhabited type, which is a fancy word for a very simple thing, a type that cannot be instantiated and cannot hold any values. Different languages implement it differently. In Swift, it’s a simple enum with no cases. In Kotlin, it’s just a class with a private constructor. The main goal is the same, it’s a type that can satisfy the compiler and convey proper semantics for things that can never happen. For instance, AnyPublisher<Output, Never> means that the publisher will never fail. Whereas Never automatically implements the Error protocol, it doesn’t implement Encodable. But it’s as easy as this…
extension Never: Encodable {
public func encode(to encoder: Encoder) throws {}
}With that very simple conformance, we could declare our GET request without problems.
extension Request {
static var allJobs: Request<[Job], Never> {
.init(path: "jobs")
}
}In fact, if you think about it, Never makes a lot of sense, as the code path that has to do with payloads (case post(_ payload: Payload)) is something that never gets executed for a GET request.
We were pretty happy with the design so far. In fact, we could say that this was the most idiomatic and Swifty way we found. Two minor issues made us look for an alternative solution though.
- We needed to create intermediate structures for most of the payloads, whereas we could simply inline simple dictionaries before.
- In order to know what the actual JSON keys are sent to the server, you need to know how
Encodableworks. The keys might be the very same property names, or maybe they are the ones in theCodingKeyenum… Or maybe they are different, depending on thekeyEncodingStrategyproperty ofJSONEncoder… Who knows ¯\_(ツ)_/¯
This made our journey a little bit longer, but it was worth it in the end.
JSON payloads as recursive sum types
A JSON, as defined in the ECMA-404 standard, can be:
- A number
- A boolean
- A string
- Null
- An array of those things
- A dictionary of those things
Only by taking a look at that definition, our intuition made us know that we were dealing with a recursive data structure. Taking into account that the possibilities were well-known and there was no point in allowing extension, a recursive sum type seemed like the most convenient solution.
enum JSON {
case int(Int)
case double(Double)
case string(String)
case bool(Bool)
case array([JSON])
case dictionary([String: JSON])
case null
}To make it work for us, we needed to have it conform to Encodable.
extension JSON: Encodable {
func encode(to encoder: Encoder) throws {
var container = encoder.singleValueContainer() switch self {
case .int(let int):
try container.encode(int) case .double(let double):
try container.encode(double) case .string(let string):
try container.encode(string) case .bool(let bool):
try container.encode(bool) case .array(let array):
try container.encode(array) case .dictionary(let dictionary):
try container.encode(dictionary) case .null:
try container.encodeNil()
}
}
}
We could then change our HTTP method, removing the Payload generic type, and have the aforementioned JSON structure as the payload of our POST request.
enum Method {
case get(_ queryItems: [String: CustomStringConvertible] = [:])
case post(_ payload: [String: JSON])
}We were able to inline dictionaries in our payloads again, making it very clear the keys that were sent to the server.
extension Request {
static func modifyJob(
title: String,
description: String,
id: Int
) -> Request<Void> {
.init(
path: "jobs",
method: .post(
[
"title": .string(title),
"description": .string(description),
"id": .int(id),
]
)
)
}
}Unfortunately, the ergonomics did worsen quite a bit, having to wrap everything in the JSON case constructors. In fact, the example above doesn’t reflect how bad the ergonomics were, especially when dealing with nested arrays and dictionaries.
"ids": .dictionary(
[
"id": .int(id),
"another_id": .string(anotherId)
"some_other_ids": .array([.int(id1), .int(id2)])
]
)Fortunately, Swift provides two really great features that we could use to fix most of those problems.
- ExpressibleBy protocols
- Custom operators
Improving the JSON type ergonomics
Having the JSON type conform to the ExpressibleBy protocols was quite simple.
extension JSON: ExpressibleByIntegerLiteral {
public init(integerLiteral value: Int) {
self = .int(value)
}
}extension JSON: ExpressibleByFloatLiteral {
public init(floatLiteral value: Double) {
self = .double(value)
}
}extension JSON: ExpressibleByStringLiteral {
public init(stringLiteral value: String) {
self = .string(value)
}
}extension JSON: ExpressibleByBooleanLiteral {
public init(booleanLiteral value: Bool) {
self = .bool(value)
}
}extension JSON: ExpressibleByArrayLiteral {
public init(arrayLiteral elements: JSON...) {
self = .array(elements)
}
}extension JSON: ExpressibleByDictionaryLiteral {
public init(dictionaryLiteral elements: (String, JSON)...) {
self = .dictionary(
elements.reduce(into: [:]) { $0[$1.0] = $1.1 })
)
}
}extension JSONValue: ExpressibleByNilLiteral {
public init(nilLiteral: ()) {
self = .null
}
}
With those conformances in place, we could then have any kind of literals in the POST payload.
"ids": [
"id": 45,
"another_id": "another_id"
"some_other_ids": [23, 56]
]It worked similar to the initial [String: Any] payload type, but with the additional type-safety.
Most of the time though, we won’t be dealing with literal values. So, even if we could have array and dictionary literals, which removed a lot of the noise already, we still had to wrap strings, numbers, and boolean values into those case constructors.
This was where the custom operator came into play.
prefix operator ^
prefix func ^ (rhs: String) -> JSON { .string(rhs) }
prefix func ^ (rhs: Int) -> JSON { .int(rhs) }
prefix func ^ (rhs: Double) -> JSON { .double(rhs) }
prefix func ^ (rhs: Bool) -> JSON { .bool(rhs) }Custom operators are a widely debated topic, not only in the Swift community but in programming in general. Some people love them, others hate them. I think that, given the right scoped context for them, they can make a great addition to your codebase, simplifying some tedious and chore tasks, providing useful syntax sugar that’s a joy to use.
So finally, having the ExpressibleBy conformances and the custom operator in place, we could have something like this.
"ids": [
"id": ^id,
"another_id": ^anotherId
"some_other_ids": [^id1, ^id2]
]As you can see, we dramatically reduced the noise that we had in the first version, taking advantage of the type system to provide the semantics and the safety that we were aiming for.
Conclusion
I still remember, quite a long time ago already, when I studied Telecommunications Engineering at university. One of the recurring topics was the Signal-to-noise ratio (SNR) for linear time-invariant systems.
I also like to apply the same SNR concept for programming, especially when dealing with types. We can think of the Signal as the type-safety, the conciseness, the correct semantics, whereas the noise would be the price to pay to have that, in terms of cumbersome syntax, workarounds, superfluous types, or boilerplate you have to create.
API design is all about trade-offs. It’s all about maximizing SNR via proper types and abstractions.
SNR is sometimes, unfortunately, quite subjective as well. For us at Jobandtalent, this playground, and the two hours we spent on it, gave us the best SNR we could have in an API client abstraction for Swift. But that’s, obviously, only our opinion.
