Scala Wishlist

I’d call myself a down-to-earth scala engineer. By that I mean that I never say words like “covariant” unless I absolutely have to. I see programming languages as a tool to make my life as easy as possible – I chose scala because it lets me get the most output for the least input. I think this is worth stating up-front, because my initial impression of scala was that it was daunting and geared toward academics, and I admit it does have some up-front costs.

What’s cool about scala – things it makes easy

Examples

def showOff(): Unit = {
   val someList = List(1,1,2,5,5,5,7,8,25,31,231,2335)
   val edgeCase = List()
   val uniqueCount = someList.unique.length
   val secondToLast = someList.dropRight(1).lastOption
   val secondToLast2 = edgeCase.dropRight(1).lastOption // None
   val res = someList.search(7) // find index of 7 in logN time = Found(6)
   val res2 = someList.view.map(x => x*x).search(100)
 // in logN time, find where 100 stacks up against this list, while only squaring logN items = InsertionPoint(8)
  val prefixSums = someList.scan(0)(_ + _) // 0, 1, 2, 4, 9...
  val groups = someList.groupBy(_.toString.length) // map into a list of 1-digit nums, 2 digit nums, 3 digit nums
  val lookup = someList.zipWithIndex.toMap // build a HashMap lookup of int -> index of last occurrence
  val lookupFirst = someList.zipWithIndex.distinctBy(_._1).toMap // build a HashMap of int -> index of first occurrence

  val countBy = someList.groupBy(identity).mapValues(_.length) // build a lookup of value -> count of occurrences  
 
  val matrix = Array.tabulate(10,10)((y, x) => y*10+x) // creates a matrix of nums 0->99
  val rowIsOdd = matrix(3).forall(_ % 2 == 1) // false
  val colIsOdd = (0 to 9).map(i => matrix(i)(1)).forall(_ % 2 == 1) //true

  val lookup = collection.mutable.Map[Double, Double]()
  def memoizedSqrt = (i) => lookup.getOrElseUpdate(i, Math.sqrt(i))
}

The above is just showing that scala has a bunch of helper functions that may seem trivial at first, but with mastery are outstanding building-blocks for all sorts of problems. Of course lodash achieves similar, for example.

Type Safety

Next the type-system. I think scala’s type system (particularly around Options and Futures) prevents a huge swath of bugs around edge-cases and concurrency by pointing out logic gaps at compile time. It takes longer to get scala code to run, but once it does it’s exponentially more likely to work correctly than node.


What I wish were changed

The gimmes

A for loop that functions like the c-style for-loop. Many linked-list and tree operations I write (i.e. leetcode) end up having to use whiles, which is blah.

An ability for a collection to have more than Int.MaxValue items. This is again something that comes up more from leetcode, but given that some collections (like Ranges and views) don’t (or at least needn’t) allocate that memory, it’s actually quite reasonable. This issue sprouts its head when you want to .search a long for example.

The culture

After the initial step of seeing map and reduce and thinking that “cool that looks handy” some of us have run into people peddling a “necessary next step” of learning exhausting thesaurus of interconnected academic buzzwords (covariant, functor, monoid…) that are really offputting. It’s been my experience that learning what a monad is has had zero effect on my productivity, and is at best a neat philosophical abstraction. Part of this complexity originates from scala’s type system. But the other half comes across as elitism and cargo-culting, fair or not.

Compilation

This is one of the real killers. I had a junior coworker who spent several hours trying to get his program to run, all while getting an illegible error. It turned out a brace was missing somewhere (or some other common mistake) which resulted in the compiler running for minutes then failing. Theoretically scala offers differential compiles, but if they are working they aren’t fast enough. Improving this alone by 75% would be more important than any and all of the fixes in scala 3.0.

Overcomplicated defaults (including libraries)

Again this is a sad one, because it would have been so easy for the normal libraries to cater toward beginners and only slowly expose complexity. I’ve seen tragic bugs (e.g. a super-fast language on a super-powerful machine misconfigured to only run 1 web-request at a time) that negate the entire value-promise of scala when put in the hands of non-experts. I’ve seen startups with hundreds of requests a day create a whole nightmare-to-debug actor-powered service because Akka. Documentation, simplicity, and guiding novices away from power-tools could have a much higher payoff than oddities like LazyList.

Also, real-world concerns (cross-future UUIDs, seeing memory usage per-thread) seem to be an afterthought, though this is true of most languages too.

The emotional tax of bad recommendations

tl; dr: I find some web 2.0 content so draining it undermines the experience of entire platforms

I think we can all relate to those moments in life when you feel overwhelmed — beset on all sides by obligations or problems. In such times I try to turn to humor, distractions, and escape.

For over a year now I’ve known that certain social-media platforms I was trying to “escape” to were actually promoting stress-inducing content that was having an entirely inverse effect. “This is my problem,” I told myself, “I’m the one choosing to open these sites.”

I never even touched Twitter. Then some news sites. Then linkedIn, nextdoor. The last stragglers are reddit, youtube, and news.ycombinator.

Reddit is the simplest illustration of this phenomenon. I have an exhausting meeting coming up, I need a 10 minute breather to clear my head, I turn on reddit for cat pictures and boom suddenly I’m witnessing a cop battering a protestor. My body has an immediate visceral response. So caught up am I in righteous indignation, that I have completely forgotten in the moment about how my goal was to relax. “I need to care about this!” screams my body.

I pruned my reddit feed, and now I’m down to “r/madeMeSmile” and “r/HighQualityGifs.” Unfortunately, if I ever log out of the site I get this default feed, which is about 25% outrage-bait (e.g. a video of a group of people confronting a car driving on the sidewalk) (e.g. “Controversial law allows police to seize and sell cars of non-lawbreakers, keeping the proceeds”).

The math here is working against me. One cute picture of a cat does not negate one horrifying video of somebody slipping on the stairs and hurting themselves. Or in the case of HN, one intense rant is not negated by one thoughtful comment.

Youtube though is the platform I’d miss the most. I seem to have no control over my youtube feed, but ideally it’d be this “Nothing political, nothing with the words ‘destroyed’/’owned’/’idiot’. Nothing about millennials/boomers. Nothing where the thumbnail is a face-palm. Nothing with the phrase ‘You wont believe'”.

I go to youtube for things like lockpicking videos, gameboy repair, primitive survival. Unfortunately, somehow I seem to get recommended a ton of Joe Rogan clips, unwanted fringe political videos, and a mix of other neutral unwanted content. And the comments are the worst.

Unfortunately what youtube’s algorithm doesn’t understand is linking one single inflammatory bad video recommendation (even if I don’t click it) may make my entire youtube experience negative. Most topics have an emotional impact on people, and web 2.0 needs to start regulating for this, or I suspect others will eventually find themselves booting platforms wholesale too.

If this were a youtube video, this is the point where I’d be telling you that this is some crisis to freak out about. But honestly it’s not really an urgent problem. I think there’s an opportunity for us as individuals to become increasingly aware of when we’re being baited/provoked and avoid platforms. I also think there’s an opportunity for new platforms or algorithms to form that prioritize giving the user the emotional experience that they are seeking.

A different github design

Earlier today HN reviewed a proposed github redesign and largely didn’t favor it.

I figured I’d take a stab at it

See the proposed before-and-after pictures above.
Here are some principles that seem intuitive to me, and maybe designers might consider them too.

  1. The visibility of an item should be proportional to its usefulness. This includes size, placement, brightness/colorfulness.
  2. There is always a flexible solution which caters to both experts and novices simultaneously.
  3. Hierarchy of UI should reflect conceptual hierarchy.

In practice here are the things I changed in my mockup, based on those 3 rules:

  • Create new file has no business being on the same line as “Clone repo.” It is a branch-specific operation next to a repo-specific operation.
  • The current-branch dropdown/button should connected-to the file-list widget. The file-list widget is showing files of that branch. The two are logically interdependent but visually separated.
  • Wiki and Insights are features I have never used on github and may never use. They should be hidden by default. They can intelligently show for repos that have ever once used those features.
  • The repo description shouldn’t be in the code tab. The repo description is an attribute to the repo itself


Good luck on your design journeys.

My Brainf Quine

A quine is something simple to describe yet surprisingly challenging to implement– a program which outputs exactly its own source-code. It is something of a rite-of-passage for an engineering afficianado. For those that consider ourselves one level beyond afficianado we always are looking to up the ante. I took two years exploratory years off after high school, and remember them fondly. Those were the days I could explore anything I wanted. Time was so abundant and problems were so scarce that I’d take on challenges like quines recreationally.

It’s a magical place to be in, when you any path feels possible and no obligations feels mandatory. It’s a time when one’s world-view is fully open, and interesting opportunities seem everywhere. It’s a time before traditional adulthood, where one can feel exhausted by unending obligations (cable bill, health insurance, change my oil, arrange my 401k, excercise more, sleep more, read more, relax more, setup dentist appointment, pickup groceries, return that item, answer those emails to those family members).

Once we’re in the “real world” it can be a challenge to remember that initial feeling of possibility. Once the lionshare of our time is spoken for, one may switch modes from expanding exploration to reduction. A mode where we filter our world into a functional place of checklists and routines to optimize staying afloat when our time, attention, and concern run short and we must ration them.

Anyways, I reminisce. But back in that era, one thing my friends and I would do is make coding challenges for each other. After a friend introduced me to “brainfuck,” a language with only 6 commands all represented as single characters, I challenged him to write a quine in brainfuck. I can see by googling that many other people like us are out there, who, like us, have been to that place where we are hungry for the next challenge to create for ourselves.

Recently I found my quine from back then it brought back memories.

 

>>>>++>+>>>>>+++>++++++>>>>>++++++>+>
>>>>++++>+>>>>>+>+>>>>>+++>++++++++++
>>>>>++>+>>>>>+++++++>+>>>>>+>+>>>>>+++>++>>>>>+++++>++++
++>>>>>++++++>+>>>>>++++>+>>>>>+++++++>+>>>>>+>++++++>>>>>
++++++>+>>>>>+>++++++>>>>>+++++++>+>>>>>++>+++>>>>>+++
>++++++>>>>>++++++>+>>>>+>+>>>>>+++>++++++++++>>>>>++>
+>>>>>++++>+>>>>>+++++++>+>>>>>+>+>>>>>+++>++>>>>>++>++
>>>>>+++>++++++>>>>>++++++>+>>>>>+>+>>>>>+
++>+++++++>>>>>++>+>>>>>++++>+>>>>>+++++++>+>>>>>+>+
>>>>>+++>+>>>>>++>+++>>>>>++++++>++>>>>>+>+>>>>>+++>+
>>>>>+>+>>>>>+++>+>>>>>++>+
+>>>>>++++>+>>>>>+++++++>+>>>>>+>++>>>>>++++++>+>>>>>++++>
+>>>>>++>++>>>>>+++>+>>>>>+>++>>>>>+++++++>+>>>>>++>+>>>>>++++++>+>>>>>+++
+>+>>>>>+>++>>>>>+++++>+>>>>>++>++>>>>>+++++++>+>>>>>+
>+++>>>>>+++++>+>>>>>++>+++++>>>>>++++++>+>>>>>++++>+>>>>>+>++>>>>>+++>+>>>>>+
+>>>>>+++>+>>>>>++>+++>>>>>+++++++>+>>>>>+>++>>>>>++++++>
+>>>>>++++>+>>>>>++>++>>>>>+++>+>>>>>+>++>>>>>+++++++>+>>>>>+>+>>>>>++++++
+>>>>>+>+>>>>>+++++>+>>>>>++>+>>>>>++++>+>>>>>+++++++>+>>>>>+>++>>>>>
+++++>+++++>>>>>++++++>+>>>>>++++>+>>>>>++>++++++>>>>>+++>+>>>
>+>++++++>>>>>+++++++>+>>>>>++>+>>>>>++++++>+>>>>>++++>+>>>>>++>++++++
>>>>>+++>+>>>>>+>++++++>>>>>+++++++>+>>>>>++>+++++++++>>>>>++
+++++>+>>>>>+>++++++>>>>>++++++>+>>>>>+>++++++>>>>>+++++++>+>>>>>++
>++++++>>>>>++++++>+>>>>>+>++>>>>>+++>++++++>>>>>++++++>+>>>>>++>
+>>>>>+++>++++++++++>>>>>+>+>>>>>++++>+>>>>>+++++++>+>>>>>++>++
>>>>>++++>+>>>>>++++++>+>>>>>++++>+>>>>>+>+>>>>>+++>++>>>>>++>+>>>>>+
+++++>+>>>>>++++>+>>>>>+>+>>>>>+++>+>>>>>+>+>>>>>+++>+++++>>>>>++++++
>+>>>>>++>+>>>>>++++>++++>>>>>+>+>>>>>++++>+>>>>>+++++++>+>>>>>
++>++>>>>>++++++>+>>>>>++++>+>>>>>+>+>>>>>+++>++>>>>>++>+>>>>>++++++>
+>>>>>++++>+>>>>>+>+>>>>>+++>+>>>>>++>+>>>>>++++++>+>>>>>++++>+
>>>>+>++>>>>>+++>+++++>>>>>++++++>+>>>>>++>+>>>>>+++>+++++++++>>>>>+>
+>>>>>++++>+>>>>>+++++++>+>>>>>++>++>>>>>++++++>+>>>>>++++>+>>
>>+>+>>>>>+++>++>>>>>++>+>>>>>+++++++>++++++>>>>>++>+>>>>>++++++>
+>>>>>++++>+>>>>>+>++>>>>>+++++>+>>>>>++>++>>>>>+++++++>+>>>>>++>+
++++>>>>>+++++++>+>>>>>>

++++++[-<++++++++++>]
<++……[-]
<<<<<<[<<<<<<]
>>>++++++
[<++++++++++>-]
<++>>++++++
[<+++++++>-]
<+>>>
[[<+<+>>-]<<
[->>+<<]>[-<<.>>]<<<.>>>>>
[-<<+<+>>>]<<[->>+<<]<[<.>-]
<<…..[->>>>>>+<<<<<<]>
[->>>>>>+<<<<<<]>>>>>>>>>]
<<<<<<[<<<<<<]>>>>>>
[<<++++++
[>++++++++++<-]>>-
[-<++>
[-<+<+++++
[>—-<-]>>
[-<++>
[-<+>
[-<<+++++
[>+++++++++<-]>>
[-<++>]
]
]
]
]
]
>[-<<.>>]
>>>>>
]

Sopping Wet — Today’s Software Ecosystem Isn’t DRY

Tl; Dr:

  • Everyone seems to understand DRY is good at the program level, but they don’t seem to understand it at the community level.
  • Examples of useless duplication include many programming languages, libraries, package managers, data-stores, tools
  • This community duplication reduces interoperability and slows productivity across the board

Section 1: Some examples

1. Why is there more than one unix/linux package manager? Do we really need a different package manager with the same commands but renamed for each programming language? Why do we need a distinct package manager for each distro?

2. Nobody seems to admit it, but Php, Ruby, Python, and Javascript are the same language, with a little sugar added here or there and different libraries. More formally, I’d say that for 99% of lines of code (written) there’s a 1:1 translation between each of these languages (on a per-line basis). I get differences like curly braces vs indenting, but it does strike me as wet that each language has rebuilt so much core functionality (date parsing, database connectivity with the use of NoSQL database solutions, html parsing, regex, etc). Wrapping libcurl, for example, is a great way to stay dry.

This leads to a scenario where “learning a language” is more about learning the libraries than anything else (e.g. “How do timezones work again in PHP?”)

3. Did MongoDB really need to exist as as a standalone application? What if MongoDB had simply been a storage engine? The concept of a datastore that adapts its schema on-the-fly and drops relations for speed is okay, but does that justify creating an entirely new data-storage technology to that end? This means millions of engineers learning a new query syntax for a potentially temporary technology. Same with the security policy, all the DB drivers. There’s no reason all the tools to get visibility (sql pro) and backup the database need to be reinvented. Plus, if it were just a storage engine, migrating tables to InnoDB would be easier.

The same point holds for cassandra (which is basically mysql with sharding and more sophisticated replication built in), elastic search, and even kafka (basically just WAL of mysql without columns). For example, a kafka topic could be seen as a table with the columns: offset, value. Remember storage engines can process different variations on SQL to handle any special functionality or performance characteristics as-needed (I do recognize what I’m describing is easier said than done, but recommend it nonetheless).

4. Overly-specialized technologies should not exist (unless built directly around a general technology). You ever see a fancy dinner-set, where for “convenience” people are offered 5 forks and spoons, each one meant to be used slightly differently for a slightly different task? That’s how I feel about overly-specialized technologies. For example, people seem to love job queues. What if all job queues were implemented on top of a SQL backend so that engineers get the normal benefits:

  1. engineers know how to diagnose the system if it fails because it’s a known system (e.g. performance issues, permissions)
  2. engineers can always query the system to see what’s happening because it’s using a standardized query language
  3. engineers can modify the system if necessary because it provides visibility into its workings
  4. engineers can use existing backup, replication, monitoring, and other technologies to store/distribute the queue (giving interoperability)

Section 2: What’s the result of all this?

  • Every time a brand-new hype technology is introduced senior Engineers are all set back years relative to junior ones (which is bad for senior engineers, good for junior engineers)
  • The ecosystem is set back as a whole (all tools, libraries that interact with the old technology are rebuilt for the new one)
  • Company is placed in an uncomfortable position because it now only has junior engineers in the given technology. When I was junior, I worked at a startup that accidentally lost most of their customers’ phone numbers because their PHP driver for mongo would convert numeric strings to numbers, and phone numbers would overflow the default integer, resulting in no fatal errors but simply negative phone numbers.
  • The company runs the risk of being saddled with a technology that will be dropped (e.g. couchdb, backbone) and will require a rewrite back to a standard technology or be perceived as behind-the-times.
  • Slow-learning / part-time engineers must keep pace with the changing landscape or face irrelevance. Those that can’t learn several technologies a year will stumble.
  • Fast paced-engineers will lose half of their learning capacity on trivialities and gotchas of each technology’s idiosyncrasies (e.g. why can’t apache configs and nginx configs bare any resemblance to each other?). Once these technologies are phased out (e.g. now it’s mostly cloud ELBs) all of that memorization is for naught. It’s a treadmill effect – engineers have to sprint (keep learning new technologies) to move forward at all, walk just to stay in place, and if you can’t keep pace with the treadmill you fall behind.Quick aside – I think things moving to the cloud is probably one of the most outstanding dev benefits I’ve seen in my life. However it continues to illustrate the point in that if different cloud providers don’t standardize then the whole development community is slowed down.

Section 3: The exceptions

There are a few exceptions I can think of when a complete rebuild from scratch was an improvement. One would be Git. In a few months, one of the most prominent software geniuses of our era invented a source-control system so superior to everything else that it has been adopted universally in a few years, despite the intimidating interface.

The times a rebuild is justified seem to be when many of these criteria apply:

  • You’re a known and well-respected name that people trust so much the community might standardize on what you make (e.g. Linus Torvalds, Google)
  • The existing systems are all awful in fundamental ways, not simple in easily-patchable ways. You’ve got the ability, time [and we’re talking at least a decade of support], money to dedicate yourself to this project (git, aws, gmail, jquery in 2006)
  • You can make your system backward compatible (e.g. C++ allows C, C allows assembler, Scala allows Java, many game systems and storage devices can read previous-generation media) and thus can reuse existing knowledge, libraries, and tools
  • You’re so smart and not-average that your system isn’t going to have the myriad of unanticipated flaws that most software systems you want to replace will. For example, angular, backbone, nosql, are all things that might, in hindsight, have been not worth learning. Of the current high-buzz languages as of me writing this (Go, Clojure, Haskell, Ruby) it’s open to speculation which will stand the test of time.
  • Your system is already built-in-to or easily-integrated-with existing systems (e.g. JSON being interpretable in all browsers automatically, moving your service to the web where it will work cross-platform and be accessible without installation)

Section 4: What can one do?

  1. Learn the technologies that have stood the test of time: linux cli, c++/java, javascript, SQL
  2. Wait years before adopting a technology in professional use for a major use-case– let other companies be the guinea pig
  3. Be judicious in usage of new technologies. For whatever reason, it’s culturally “cool” to know about the “next big thing,” but it’s better to be late and right than early and wrong.

Our obligation as leadership

The talk of importance of values is one irony of the San Francisco scene, if not human nature. The same values are discussed everywhere; so why then is it that these same values seem to be applied nowhere?

Could it simply be that it’s much easier to see the mistakes of others than our own? Perhaps those of us in positions of power often subjected to less scrutiny? Yes, to both of these. And so it becomes our own greatest personal challenge to remain true to our goals when nobody else is checking.

Let’s consider the value of ownership. What does this mean for us? It’s easy to look at the engineers who report to us and think about the times they didn’t take a personal investment in what they were doing, and how that harmed the company.

But for us to be good at our job, we must challenge ourselves to hold ownership, because it’s rare somebody will tell us when we’re not.

So what does ownership actually mean? Well, if an engineer is taking ownership of a project that to me means that she takes personal and emotional accountability for doing the best feasible job she can at it. If it’s broken on production, she is treating the lost revenue like her own.

But what does ownership look like in a manager? To me ownership is no less of an obligation. In fact we have more obligation because we have more influence. We should hold ourselves personally accountable for accurately assessing the merits of our direct reports. If a great manager screws up, he should lay awake at night until he fixes it, just as a great engineer would wake up to fix a production issue. A great manager won’t “good enough” it and wait until the next review cycle to compensate. Us not admitting to an error to save face is no less excusable than an engineer covering up when he breaks the app (which is to say absolutely inexcusable).

Ownership is about caring about the job getting done correctly, at a core level, above-and-beyond what is immediately asked of you. If you see a problem that nobody else sees, ownership is taking it up and ensuring that it gets resolved, regardless of how it reflects on you. Ownership is helping the company and the customer, even if it costs you your job (be that whistle-blowing, disagreeing with an incorrect authority, refusing to do something immoral/illegal).

If you institute an initiative that is clearly ineffective, then you should admit it and withdraw the initiative. Your self-promotion is not a contribution to the company. If you are a great manager, you won’t make it your direct reports’ job to convince you they are great; you will make it your job, your contribution to ensure everyone beneath you is being used to the best of their ability. And if your manager is great, she won’t expect you bring donuts, wear a tie, show up early, or flatter; because it will be her contribution to the company to accurately evaluate your work (and not how much she loves or hates you).

And if, when you hear this, you find it mildly irritating that anybody would ask so much of you… then don’t be surprised when those you lead act as do and not as you say. Your attitude is the irony of the SF tech scene.

Connect 4 – illustration of Matrix rotation in js

Once in an interview, I was asked to determine whether a somewhat filled in connect 4 board has a winner. Here’s an elegant solution using an arbitrary rotation function: Rotate the board 0°, 45°, 90°, 135°, and check for 4 in a row in each of those rotations.

Since I couldn’t find a good rotation solution online I decided to post a solution here (javascript):

var _ = require('underscore');

var rotateArray = function(array, degrees) {
    var radians = degrees * Math.PI / 180;
    var round = (f=>parseFloat(f.toFixed(13)));
    var coordinates = _.flatten(array.map((row,y) => row.map((value,x)=>({value,x,y}))));
    var translatedCoordinates = coordinates.map(spot=>({value: spot.value, x: round(spot.x*Math.cos(radians) - spot.y*Math.sin(radians)), y: round(spot.x*Math.sin(radians) + spot.y* Math.cos(radians)) }))
    var rows = _.chain(translatedCoordinates).groupBy(x=>x.y).values().sortBy(x=>parseFloat(x[0].y)).value();
    return rows.map(x=>_.pluck(x,"value"));
}

var hasFourInRow = function(array){
    var vectors = [0, 45, 90, 135];
    var matrixToString = (m => (m.map(r=>r.join("")).join("\n")));
    var versions = vectors.map(degrees => rotateArray(array, degrees)).map(matrixToString);
    return _.any(versions,(b=>b.indexOf("****")!==-1));
}

And to illustrate the use, here’s sample test cases:


var testCases = function(){
    var win1 = [['*','*','*','*'],
                [' ',' ',' ',' '],
                [' ',' ',' ',' '],
                [' ',' ',' ',' ']];
    var win2 = [['*',' ',' ',' '],
                ['*',' ',' ',' '],
                ['*',' ',' ',' '],
                ['*',' ',' ',' ']];
    var win3 = [['*',' ',' ',' '],
                [' ','*',' ',' '],
                [' ',' ','*',' '],
                [' ',' ',' ','*']];
    var win4 = [[' ',' ',' ','*'],
                [' ',' ','*',' '],
                [' ','*',' ',' '],
                ['*',' ',' ',' ']];
    var lose1 = [['*','*','*',' '],
                 ['*',' ','*','*'],
                 [' ','*','*','*'],
                 ['*','*',' ','*']];

    console.log([win1,win2,win3,win4, lose1].map(hasFourInRow));
    //outputs true, true, true, true, false
}

testCases();

Stories, and Meaning [at work]

Tl; Dr: All people are motivated by meaning of their work. The ability to shape the “stories” that we tell ourselves gives meaning and thus is an essential motivational tool. This meaning is also key for personal satisfaction.

Let’s consider a standard engineer. There are a lot of “outlooks” this engineer might have. I’ll go over 5 out of thousands of possibilities just to illustrate some variety:

  • I’m the star engineer here, and I’ll turn this place around. Others might not quite see it yet, but it’s only a matter of time.
  • I’m a great engineer but nobody appreciates my work. How come that other engineer got promoted, I don’t think their work is as good as mine. I should run the company, the CEO is only CEO because he has blue-blood 1% investor friends. The world is rigged against me.
  • I’m so excited to be an engineer. I earn multiple times the median household income and I get perks too, while my other friends are still paying off their college debt.
  • Work’s good. It’s just a job though.
  • Why am I always the one under pressure? Nobody ever thanks me no matter how hard I work. Why do I get mistreated? I’m such a victim. It’s not right.

What I’m proposing is that the same engineer having the same experiences could retell that same experience in any of the ways I listed above.

In fact, you’ll find that a lot of people gravitate towards common narratives. I have a friend, let’s call him Fry who always sees the same story: The big guy is mistreating the little guy. One thing I learned quickly was that he liked and actively chooses (if not entirely consciously) to see the world in these terms. And if I want his help, I can motivate him by giving him a rebel story he can play a part in.

For example, if I’m trying to create a new tool, I could tell it in a way that plays up the opposition, especially if it’s an entrenched “senior” group that’s cautious about new ideas, and it’ll make him want to help.

Everybody has a narrative or two that resonates with them. I have another coworker, Bender, who always promotes the story: People are stupid. I know not to try to change his story, because it’s a choice and likely inspired by some very frustrating life experience (I imagine those frustrations are valid, albeit rather long-lived). If I need him on a PR I can ask it in a way that implies I need his expertise. That way he act can out his fantasy of undoing the damage of incompetent/indifferent through the act of improving my code. A win/win.

Sometimes these narratives are just plain counterproductive. I can think of experiences when teammates were annoyed that some “undeserving” party “stole credit.” In these situations you can try to rewrite the narrative, though it takes a certain skill. One way to accomplish this is to bring up an opposing narrative like, “Yeah so his name was mentioned in a meeting big deal… I don’t do great work so some manager can mention my name, I do it because there are literally millions of users who are experiencing what I make and I think I’m lucky to be able to be in a position to help so many people. That’s something I can be proud of.” [And as an aside their is no duplicity in this. It is our choice whether to aim to “do good” or to accrue material. Each philosophy has its upsides.]

And the fact that people are choosing to live these stories is an important fact. For a long time, I would have thought doing this type of thing was “manipulative” or “tricking” people into work when I could instead explain to them why their underlying narrative was too black&white. But knowing that these narratives are choices, with symbolic importance, that give meaning to the individual who holds them, I see now that people choose to live these stories hoping to “play them out.”

And most importantly, we all have narratives. Mastering other peoples narratives is a great tool, but mastering your own is probably more important.

———

Part 2.

If you’re a leader in an organization, you need to be aware of the stories that occupy the minds you oversee. Do people see themselves as battling each other for recognition? Do they see themselves as allies against a great evil? Do they see themselves as victims of your rule?

People within the company all have their narratives, and those attitudes are contagious. Your actions will have a great role in deciding which attitude wins out.

I’ve seen organizations torn apart by leaders who were out-of-touch with the effects of their actions. The cultural effects you have are of supreme importance. Actions like requiring engineers start an hour earlier, for example, are incredibly dangerous, because they give room for the narrative “We are seen as code monkeys” to thrive.

To win this battle you must understand the narratives that you are battling against. You must truly understand them, you must know and appreciate the day-to-day of your workers. For example, you must know that your Office Manager Lela is frustrated, sleep-deprived, is wondering where her life is going, and is starting to feel like the “manager” in her title is meaningless. When you know the people you work with you’ll know what stories are compelling to them. When you understand, you will be capable of offering the service of arming them with a better story, a brighter story, a more compelling story. Because as bad as Lela’s life is, it’s better for everyone if she feels like her work matters at the end of the day.

This is your value.

The Professional Web Codebase

One of the major advantages of the high career mobility in SF … Cross pollination. Yet each new codebase I comes to seems to lack a few of the following. So I figured I’d list out the essentials that every web codebase should have.

  1. The Obvious: Git, IDE, MV*, relational DB, 3 environments that match production.
  2. Language regularization:Each language has its own idiosyncrasies. Learn your languages weakness and make helper functions to accommodate. In PHP, these weaknesses include single-threaded, bad undefined index default behavior.My preferred solution to PHP’s single-threaded limitations is to make use of a job queue. This will emulate additional threads for heavy tasks (e.g. sending emails to all of your user base) or high-latency tasks that ideally should be asynchronous (e.g. certain API calls). I’ve seen this done well, and I’ve seen it done poorly. From experience, I’d point out the following lessons: don’t use a message queue as a job queue and the DB is a great place to store a home-grown job queue [because it provides great visibility, resilience, can be controlled programatically, can be exported to disk, is atomic, etc].

    In PHP, out-of-bound reads on arrays return null and log a notification to the event log. As this is never the engineer’s intent, I always write two helper functions. One that looks up an key and returns a specific value if the key isn’t found (for cases where you’re using the array as a set). The other function looks up the key and returns an exception if the key is not found (for normal cases where the key should always be present).

    I strongly believe once you handle these edge cases most popular languages are pretty interchangeable (javascript, java, php, python, ruby).

  3. A powerful, generic staging environmentOne of the most impressive and useful technologies I saw was something we called humblebrag. It was a script that caught all requests to *.company.com/…. It then took the *, checked-out the corresponding git branch, and mapped the request to that particular branch. Thus in effect, we had a zero-effort way to have all branches usable simultaneously on a single server, even by non-engineers.Doing this can be a little harder in more complex environments with versioned services listening on ports, if you don’t plan for it early.
  4. A circuit breaker. Download one from github.
  5. An ability to do cross-server mutexes.
  6. A simple, generic read-through-caching function.
  7. A powerful logging system connected to an alerting system
  8. An A/B test system. Even if you’re not doing A/B tests on user-experience, it can be a great way to rollout to a small portion of users or internal-testers to ensure stability on a new environment.

 

 

Understanding engineers by understanding authors

I’ve written before about the challenge it can be for non-engineers to understand engineering work. I’m not the the first to observe that the non-technical need a metaphor to understand invisible and often abstract output an engineer creates. Traditionally, that metaphor has been physical engineering/construction. Others have already, and rightly, criticized the physical labor metaphor.

Instead of ridiculing the wrong metaphor, I offer a substitute. I posit this: To understand engineers, think of them like writers.

to understand engineers, think of them like writers.

Why is this a good analogy? It explains several aspects about engineering that otherwise seem downright unreasonable.

  1. Emotional attachment to work
    Like an author, many engineers see a work project as an opportunity to exercise their creativity and build something the take pride in. Like an author, engineers develop a sense of owning what they create. If the project gets cancelled, handed-off, or drastically changed (especially without notice), compromising their artistic vision, the engineer may experience frustration. But this is natural, how would an author feel in such situations?
  2. Difficulty in measuring progress and estimation.
    If software were physical construction, it might be reasonable to have an easily-predictable timeline. One might confidently answer “how far along are we?” But like writing, and unlike construction, everything one makes in software is new. And like writing, progress can’t merely be measured in the amount typed, refinement and reduction are actually an important and lengthy part of the process.
  3. Ambiguity and subjectivity of excellence
    Every week somebody writes a blog post on how to find out in an interview who’s a good engineer. And every week it gets (rightly) torn to pieces by commentators. The fact of the matter is that there are many independent components that comprise engineering talent and measuring them is very difficult. Even among the very best stylistic difference might mean two greats may not appreciate each others’ virtues, like Faulkner and Hemmingway.
  4. Challenges of collaboration
    And on that note, collaborating can be hard. Like writing, there are an indeterminate number of ways to write software, but each engineer has a style. I can tell you from personal experience that writing software with other people is hardBoth want a sense of ownership and freedom to express their creativity, but must now answer to the confines of the project itself as well as their partner (or team’s) creation. Imagine a group of professional writers trying to work together, of varying cultures, varying talents, varying dispositions, and varying skill-levels. With no objective answer to many issues, engineers reviewing one another’s code is a touchy topic.
  5. A need for freedom
    As a creative endeavor, software’s workflow can be unpredictable. A creative solution may come at any moment, often in the shower or a dream. Sometimes, when all the factors align, answers comes pouring out at a great pace. During such times, the absolute worst thing that could happen would be an interruption (or a required meeting). Compromising on artistic process will compromise the product.
  6. It’s not work
    One of the most persistent and most damaging misunderstandings I have witnessed is the erroneous perception that engineers will avoid work if they can “get away with it.” Perhaps those who feel this way think of engineering as labor, as if it were physical construction. But writing code, like other writing, is a passion that cannot be avoided. When I go on vacation from work I always end up recreationally writing code within a week. I can’t generalize for every engineer, nor every project. Like a writer, if an engineer really doesn’t believe in her work for moral reasons or is so constrained she has no space for creativity then she may lose motivation.
  7. Can we rewrite?
    Another commonality is that due to stylistic differences, every individual prefers their own way of writing things. (And to be clear, in software there is a lot of room for styles, styles that are often so unique that one can often tell who wrote code just by reading it). Each considers her own style clearer, cleaner, or better; often because it aligns so directly with her own thought process. Learning to compromise on this a key skill.
  8. Some produce more than 10 times another
    Is there any number of E.L. James that could produce the Great Gatsby? For specialized work (e.g. advanced problem solving, revolutionary user experiences) there’s just no substitute for a prodigy. On the other hand, if what you’re making is the software equivalent of a gossip magazine, having Shakespeare on your team might actually be a recipe for failure.

I want it to be clear that I’m not saying all engineers automatically deserve to have any project that they work on adapted to conform to their artistic whims. I am advising all to be sensitive to the pride and passion that people put into work.

But most of all I’m trying to help people understand engineers. It can be hard to relate to an engineer who expresses slight negativity about meetings, blows off people who check-in on them, never knows how long their work will take, doesn’t like standard schedules, might prefer to work alone, gets disappointed when their work is shifted, or thinks they’re amazing. So if you care to understand and relate, I’ve given you the tools.