Category Archives: Uncategorized

Conference Summary: Winter Tech Forum 2016

I spent last week in Crested Butte, Colorado, for the Winter Tech Forum. I’ll post here about some of the sessions I attended. We continued our conversations while skiing (cross country, for me).

(Also, see Drew Stephens’ summary here)

Who attends?

This is a conference that emphasizes a personal connection between people. The relationships built at the WTF extend long beyond the week of being together. This was the 10th anniversary of this conference (born as Java Posse Roundup), attracting a dedicated group of “regulars” each year and a healthy group of newcomers as well. This year was no exception. I think that we had about 35 people total, including about 8 that were totally new. Several of those people came with coworkers who had attended previously, while a few came entirely on their own. And two guys joined in the middle of a week after a barista found out that they were developers and asked if they were in town for “the conference”. Yes, it’s that kind of a town.

An Overview

The conference is a self-organizing conference. We met on Monday morning to put some sessions on “the board” and to discuss the overall conference. There are three 1 hour sessions each morning, determined by attendees, and then people get together for lunch and afternoons to either code, ski, snowshoe, build a lightning talk, or whatever else makes sense. Lightning talks were scheduled for Monday, Tuesday, and Thursday, so we got back together for dinner and evening sessions each night. I’ll describe it all below …

Sessions I Attended


I went to “Is Java 8 Functional Enough to Stem the Flow?“. We talked a lot about alternative functional languages, and the new features in Java 8. One of the conclusions that I reached was that Java 8 seems to be taking people away from Groovy more than from Scala. And Kotlin also seems to be stealing away Groovy developers. We didn’t hear a lot about migrating code *to* Java 8, but rather that the rework cost was high, mainly conceptually, especially when moving something from Scala to Java 8.

I also attended “The Next Step to Creating a Trust Organization”, which was about teal organizations. One question I’ve always had about teal organizations is how are they funded? Realistically, someone is plunking down cash. Unless everyone is “in” financially, how can the organization truly be teal? Bruce described an interesting concept whereby the original investors “buy in” with cash while employees “earn in” to the point where the finances eventually equalize. This was enlightening to me. We also discussed a new legal entity, called a B Corp which seems to be more amenable to this structure than the C or S or LLC ways of incorporating a business. Teal organizations use “advice process” to gather opinions about a decision to be made, and there’s peer influence on those decisions. Good decisions (for the business) are rewarded by the peers, while bad decisions may ultimately land you on “the beach” (ahem, voted off the island). I’m sure it’s much more complicated than this, but it was an interesting conversation. I can see how little decisions, that edge the company forward, can be easily made this way. I’m left wondering about big, disruptive change (e.g., Netflix moving to streaming and AWS)? Will an advice culture have the appetite for this? When a cross functional, expansive change is needed, who will “lead” the charge?

For reference, in the competing slot was Fluid and Flexible REST API.

After a lunch of talking about the sessions we attended, I headed off to cross country ski with some other attendees. We went to the north end of town, past the yurt and out to Mike’s Mile. Great ski, but a little sticky due to the fresh snow on top of the wet snow underneath (it was warm!). While I was skiing, several others were hacking on projects or working on the concurrency chapter for Bruce’s latest book.

After dinner, we reconvened for lightning talks. Here’s the Monday list. As you can see, it’s a mix of technical and nontechnical topics.

  • Skimo for Noobs (James Ward)
  • My adventure with nonviolent communication (Bruce Eckel)
  • Finch: A Scala combinator library for building Finagle HTTP services
    (Chris Phelps)
  • What hipchat can tell you about coworkers (Octavian Geagla)
  • Entropy: resistance is futile …or is it? (Julie Pitt)
  • Liquid Software )Fred Simon)
  • How to not do skimo and what it can teach you about software (Octavian Geagla)
  • The racing rules of sailing (D. J. Hagberg)
  • Drones for fun and profit (Chris Marks)


I started out the morning at a discussion about Immutable Infrastructure, something that is near and dear to my heart. While we were discussing AMIs, containers, and other approaches, there was a conversation about “Seeking failure as a counter-philosophy to demanding perfection” and another on “Starting a coworking space”. 

That session was followed with Graph Databases: Do I want one? and  When less is more – close shaves with Occam’s razor. I attended the latter, which was a fantastic discussion on MVP, paring down initial product offerings and experimentation.

And for the third session of the day, I had to decide between Testing in a Crisis and Adopting agile inside organizations – State of Play of software planning, delivery and adaptability. I chose “Testing in a Crisis” and it led to discussions about Chaos engineering at Netflix and I ended up staying back during the “afternoon session” to pull together a lightning talk on Chaos. Other lightning talks on Tuesday evening included. In some cases, I only included the speaker, because I didn’t have a copy of the title …

  • Funny business with online reviews (Gordon)
  • libGDX (Chris Z)
  • Jupyter AKA IPython Notebooks (Jack)
  • “Knitting, Yo” (Joey)
  • Barbeque. Texas Style (Rob)
  • Octavian Geagla
  • Russell’s Paradox and the Y Combinator (Marshall)
  • Peter Pilgrim
  • A View into Chaos (Dianne)
  • Why your co-worker might not be a jerk (Andrew Harmel-Law)
  • Everything. And a Pony too. (Carl)
  • CrapCan racing (Drew)


Hackday is on Wednesday, where groups form to “build something” or work on a project of their choosing. Some people work on things for their jobs that they never get around to, while others do something entirely out of their domain. The hackday projects are presented in the evening session on Wednesday.

This year’s hackday presentations included:

  • Whiskey as a Service (a crowd favorite!). See a wiring video here. And the demo here.
  • Port HdrHistogram to Rust to explore borrow checking and concurrency
  • Build a distributed system with Kubernetes
  • Explore Pony and build something cool
  • Slackbots for great justice
  • Finding a better solution for open spaces in the cloud.
  • Bring women back to Computer Science. Let’s design a curriculum for computer scientists who have left the field for a few years, using existing sources and creating new ones.

In previous years, we have recorded hackday presentations and lightning talks. I think we should do that again.


For the first session on Thursday, the attendees chose from:

  • Spark:tips, tricks,best practices
  • Is slack the new office?
  • Signal and noise – metrics, testing, and other risk reduction magic sticks

I went to the Signal and Noise session. It was quite interesting, discussing the metrics that we all capture, how we act on those metrics, and the danger of becoming myopic about metrics. We also discussed the nirvana of having a great business metric to drive behavior, and the importance of limiting the times we page our on-call to true problems, avoiding false positives.

The second session of the day offered:

  • Fighting 2 monsters: morale vs. delivery
  • Cloud Event Handlers (AWS Lambda, Google Cloud Dataflow, etc) WTF?
  • IoT: Peak Hype? (the “peace dividend of the cell phone wars)!
  • Getting women back in tech

The Getting Women Back in Tech discussion was an offshoot of my lightning talk the night before and we went in a lot of different directions. If women are under-represented, black men and women are virtually invisible, being hired at a percentage below their graduation rate (where are they going?). We also talked about how to know your top of market and reaching out into your network for advocates and mentors. We think that women who left the field with strong CS fundamentals could be encouraged back with a refresher plan (assuming skills need exercising and some tooling has changed) and an interest in bringing them back. Look for more work from me on this in the future. I’m testing some courses and trying to devise a plan, with a little help from my (male and female) friends.

We wrapped up the morning session with a spirited discussion on How can we interview/hire better?, where we explored techniques companies use for screening, hiring, and sourcing candidates. In a competing room, people discussed Are standards still relevant/open source governance.

On Thursday night, we have a tradition of the Progressive Dinner. Since most of the attendees tend to share houses with other attendees, the Progressive Dinner gives us a chance to visit many of the houses and sample our peers’ cooking. We started out with paleo lasagna, then steak and veggie kabobs, moved on to burritos, then sweet potato/sausage hash or veggie hash, and ended with paleo chili. Each house had a 30 minute window, and the movement encourages new discussion groups to form. Did I mention that the food was great?!

We wrapped up the evening at the conference center, with dessert (including bread pudding!), and with more lightning talks. We found that the Monday/Tuesday lightning talks get filled up by those who have attended before, but some folks need a little more time to become comfortable with the group and to settle on a talk. And they were fabulous!

  • How this conference made my life better (Daniel)
  • Kotlin (Nadav)
  • What I learned about Ember.js in 24 hours (Gordon)
  • Coordinated Omission (Marshall)
  • Things I learned from being a musician (Gordon)
  • Cucumber BDD (Chris and Jack)


Friday was the official wrapup of the conference, with the last session (an all attendee session) ending around noon. The choices for Friday included:

  • Cross functional teams/squads/feature teams
  • Is there room for functional languages/clojure/Pony?
  • Retrospective on experiments at WTF 2016

I convened and attended the Retrospective session. We had tried several experiments, including moving from a physical board to an online board, and using Slack for communications. Almost universally, we disliked the online board. We felt that we lost some of the communication that we had previously achieved by standing in front of a physical board and discussing the sessions during the breaks. We did see some benefit in being able to see the schedule from everywhere, so we will try a more hybrid approach next year.


  • Wargames and Chaos
  • Creating an intelligent system that’ll manage our infrastructures.
  • How to help foster a culture where people want to share public data in open data and software platforms?

I attended the Wargames and Chaos session. Everyone is at a different place in their journey toward building resilient systems and testing through Chaos. It was interesting to discuss the various approaches.

The conference concluded with a general session where people talked about what they learned, how they benefited and ideas for the future.

In the evening, we held a wrapup (catered) dinner at the local museum. The historical building was interesting (including a model of the town and a train!) and the food was great. Historically, the Friday dinner has seen less attendance, as the adventurous souls have cross country skied or snowshoed out to the Magic Meadows yurt for a catered dinner, while others have chosen Djangos, a wonderful restaurant on the mountain that shares its name with a Python web framework. It was great to have everyone together for dinner this year (but I personally love the yurt).


I said that Friday is the official wrapup, but the unofficial wrapup occurs on Saturday morning, when most of the attendees who are still in town show up at Bruce Eckel’s house with the leftover food from the week and the attendees use his kitchen to turn the food into an amazing breakfast. This year, that meant steak, potatoes, eggs, fruit salads, vegetables, and lots and lots of bacon. People depart throughout the morning and afternoon for various flights or in a convoy to Denver (about a 4 hr drive).

Developer Retreat

This week, the conference didn’t end abruptly. Instead, Bruce is following it up with a Developer Retreat for which a few folks stayed. The retreat is even less structured than the conference, and it will be great to see what they do!

Join us next year!

If you need a surge of technology, or a burst of energy built from talking about new ideas and meeting new people, consider attending next year. Look for an update on the Winter Tech Forum page.

DevOps at Netflix: Posted and Upcoming Talks

I’ll be talking about DevOps at Netflix next week at JavaOne. In the meantime, I wanted to highlight some amazing talks that are already out there that go deep in some areas that I won’t.

In particular, check out these talks from AWS:Reinvent.

I hope to see some of you next Tuesday (11 am) or Wednesday (10 am) in San Francisco.

Highlights of Grace Hopper Celebration of Women in Computing 2015

I’m writing this on the last day of the Grace Hopper Celebration of Women in Computing. I truly wanted to attend my co-worker Lilit Yenokyan’s talk today, A Fine Line: Balancing Motherhood and Career, but instead I’m actually making good on that advice and heading home a day early because I haven’t been home all week.

I cannot express how inspiring it is to have spent the week with 12,000 women in technology. In a field that is still horribly underrepresented by women, it’s absolutely critical to bring together this critical mass to visualize those who are in the field. Maybe a picture will help …


Also at the conference, Netflix sponsored a Professional Development Leadership Workshop focusing on Showcasing your Work, specifically giving your first lightning talk. I had the privilege of introducing the topic, which is near and dear to my heart. I have both attended and presented lightning talks and I think that they are a great way to get people interested in something you’re passionate about. Getting more women to showcase their ideas and to get on stage is one of the motivations for the conference, and I’m glad that Netflix was able to participate in this way.

There were many great talks and keynotes, but the highlight of my week was Sheryl Sandberg’s keynote. I can only describe it as amazing, inspiring, and thoughtful conversation. The interview that followed with the incredibly talented and funny Nora Denzel was insightful and fun.

In both, Sheryl offered 3 pieces of advice to the audience:

I’ve done the “Three Things” on and off, and I think it’s definitely worth trying.  I suspect one of my coworkers from Netflix has already created a circle (can’t wait to get back to see!), and, of course, I agree with her advice to stay in tech. I’ll mirror what Sheryl said: Our jobs are impactful, well-paid, and flexible.

And men, here’s a call to action for you. If you’re in tech, I hope that you will encourage your wives, sisters, daughters, nieces and friends to go into computer science or stay in the industry. There are more of you, so the numbers game says that your involvement will increase our numbers faster. I hope you’ll help.



Why is Netflix at Grace Hopper this week?

If you’re at the Grace Hopper Celebration of Women in Computing this week, you’re not alone. There are 12,000 people here! Wow! What a difference from the Michigan Celebrates Women in Computing conference I attended several years back, held at a remote Michigan State Biological station.

But — why is Netflix here? Why are other tech companies here?

Our industry is at a crossroads. We all compete for great talent to solve the interesting and difficult technical problems that we all face. And when we don’t reach women, we make this problem even harder. Why have women been underrepresented in our field? Perhaps women haven’t embraced the field because we haven’t accurately represented what it means to be a computer scientist in 2015. And it’s our job to help grow that awareness. I want women to know what it’s like to work at Netflix, of course, but I mostly want women to know the wide variety of problems and the diverse career that computer science can offer.

So — that’s why we’re here. We want to understand how the 12,000 women who are here are thinking about our industry. And we want to support them in their career growth and aspirations. And we even have a few really cool hoodies to give away.

Netflix is sponsoring a Professional Development Leadership Workshop today (Wednesday), from 10:30-12:30 and from 1:15-3:15 in the Grand Ballroom C, on Level 3. The topic is “Showcase Your Work: How to Give Your First Lighting Talk”. Please join us!

Michigan, Michigan Tech, and Netflix … oh my!

I was thrilled that the Michigan Tech Spring Break trip to Silicon Valley was able to make a stop at Netflix. My co-worker, Roy Rapoport, and I had some fun conversations with the students and I’m looking forward to their LinkedIn requests when they get settled back into life in the Copper Country. Oh, and fun tidbit — MTU was recognized in the “Top 20 Public Colleges with the Smartest Students” by Business Insider this week. Congrats to my alma mater!

And, this week, Evan Hauck stopped by Netflix to visit. He’s in town from Southwest Michigan to speak at the GPU Conference in San Jose. I first met Evan when he spoke at CodeMash Conference, two years ago. He’s doing some really interesting work on mass spectrometry while co-oping at Leco Corporation in St. Joseph, MI. What’s so unusual about Evan is that he’s a high school senior this year. A-mazing. Oh, and the connection with the Michigan Tech blurb above? Evan will attend Michigan Tech next year, and will study Computer Science. That “smartest students” thing is definitely getting a boost from Evan. Looking forward to seeing him on the Spring Break trip in a few years!



Fun at Netflix … join us!

My team is still looking for some great developers who feel passionate about working with other teams, improving resiliency in the cloud, and building out a continuous delivery platform. You may have heard of the Chaos Monkey — we have many more ideas where that one came from!

If you have a solid Java background, we definitely want to talk to you. Lots of JVM language excitement on the team: Groovy, Scala and yes, some Java too. Our jobs site is at

This is an amazing time to be at Netflix. I hope you’ll consider reaching out and talking to me about our team. I’ll be at SpringOne2GX in a few weeks (9/9-9/12), in Santa Clara. We’re also hosting a Scala meetup at Netflix on 9/9.

You can find me on twitter at @dmarsh or on Linked In. Hope to talk to you soon!

Conformity Monkey: now open sourced

This week, Netflix open sourced yet another member of the Simian Army: Conformity Monkey. This monkey is charged with inspecting Amazon Machine Images (AMI) to determine if they meet the rules that we have specified.  At Netflix they range from ensuring that security guidelines are met to specifying failover information.

By open sourcing Conformity Monkey, Netflix has made this software widely available so that your guidelines can be specified as rules in your configuration. As standards emerge for cloud deployment, it can be a confusing place for new developers. The use of Conformity Monkey can help structure your environment around emerging best practices, developed internally or in the community as a whole.

Let us know how you might be able to use Conformity Monkey by commenting on the announcement post. We would love to hear from you.

Netflix is hiring for the Simian Army team. Join us to work on the monkeys!


Open Source Software at Netflix

Netflix has decided to open source many projects, contributing to the mindshare for cloud development. Follow the Netflix Tech Blog for the latest news, but I’ll also highlight things that I think might be interested as I have time.

The Simian Army is Netflix’s solution to keeping your cloud working well. From the Chaos Monkey (which improves your overall, long-term resiliency by shutting down your instances) to the Janitor Monkey (which detects and cleans up instances you no longer need), the Simian Army is worth investigating if you use the Amazon cloud. It’s available as open source, so you can use it and even choose to contribute.

Today, another team at Netflix open sourced Garbage Collection Visualization (gcviz). Being able to look at garbage collection as events is essential to understanding its impact on outages.

Oh, and if you REALLY like the Simian Army, maybe you will come join my team. We’re hiring for work on the Simian Army and Edda.

Greetings from the Strange Loop Conference in St. Louis

Strange Loop is a conference held in St. Louis, MO, started by Alex Miller. Much like CodeMash, Strange Loop is a developer-organized conference, and is offered at a price (around $250 for 2 days) that attracts those who self-pay and those who work for small companies. I’ve found that the self-motivated individuals are engaged attendees! I decided last winter that I wanted to branch out to attend other regional conferences beyond CodeMash and Strange Loop was top on my list. I was thrilled when Alex asked me to present the Scala Koans at Strange Loop as a 3-hour workshop on Sunday.

The koans approach to learning computer languages offers small exercises, in a test-driven manner, so that people can learn a language by through small steps and self-discovery. Offering a koans workshop is an effective way to encourage people to work on the exercises, since they can ask questions and stay engaged. Particularly while the koans are being developed, any gaps in our “lessons” can be addressed by the instructors, on site. Strange Loop targets 30-40 attendees for workshops, so I asked Joel Neely and Daniel Hinojosa, both who have experience with the koans, to co-present at Strange Loop. This offered a 10:1 student to instructor ratio, which ensured that people were able to make good progress in the 3 hours. (Having 3 presenters also allowed one of us to slip out and order some pizzas for our hungry students, since our session ran 11:30-2:30 and hungry brains don’t focus well! And no, we didn’t plan to order pizza ahead of time!). Our session was well-attended and we got some great feedback. Hallway rumblings and tweets seem to indicate it was well-received. For those who are interested in learning more about the koans, I’m in the process of bringing up a website to provide resources, code, and other hints at It’s not live yet, so I’ll let you know when it’s up (expecting in the next week or so, depending on how much time I can find to get the content there). In the meantime, you can access the student exercises at our bitbucket site.

Back to Strange Loop. The workshop day was an optional day. The first full day of the conference was on Monday. I attended some interesting sessions, including:

I also attended a purely fun session, “Learn to play Go” by Rich Hickey, creator of Clojure. Finally, I may be able to figure out what to do with the Go board that’s been sitting in my basement for a very long time! I was able to play a game with another newbie and we were very evenly matched!

The Scala talks were interesting. I tend to focus on the simplicity of Scala, as a better language than Java for the JVM. These talks were focused on getting the most out of the functional aspects of Scala. As the industry sees momentum toward using functional for what it does best, this will be very relevant. Strange Loop, in general, has a fairly functional bent to it, and that’s quite fun!

I’ll fill in about day 2 of Strange Loop later. But there are more talks to attend!