Joe Hillenbrand

Software Engineer
Seattle, WA


Infrastructure Developer @ elastic (2014 - 2017)
Maker of Elasticsearch, Logstash, Kibana, and Beats (Remote)
The role of the infrastructure team at Elastic is to develop and manage internal and external services and act as infrastructure consultants for the product developers and other employees. We provided developers with infrastructure solutions and tested the company's products by using them for these services.
  • I acted as the primary software developer for the infrastructure team.
  • Developed a configuration management library in Haskell, named Craft, as a more versatile and type-safe alternative to Puppet and Ansible.
  • Built, with Haskell/Craft, a large scale automated test suite to deploy and test Elasticsearch in 100 to 1000 node cluster configurations on AWS and GCE.
  • Developed, using Haskell/Craft and oauth2_proxy, a highly available proxy service to allow employees to give secure Google SSO access to their internal Elasticsearch clusters without installing anything, having any special access, knowing any secrets, or asking the infrastructure team for help. It has a management portal and would automatically create user specified domain names and Let's Encrypt certificates.
  • Deployed the Vault secrets management service in a highly available configuration.
  • Wrote a custom wildcard DNS service ( in Haskell as a highly available alternative to or The existing solutions were not reliable enough.
  • Built AWS Lambda functions to monitor the company's internal AWS usage and billable activity.
  • Various other development in Python, Bash, Clojure, and Go.
Backend Software Engineer @ Flowroute LLC (2013 - 2014)
VoIP/SIP Trunking Provider (Seattle, WA)
This was my first role as strictly a Software Engineer rather than as primarily DevOps or Systems Administrator. I was the company's first full-time developer after the CTO. My work was centered around all the backend services. Because the company is a service provider, we were motivated to provide highly available services by building a fault tolerant and distributed architecture.
  • Developed an asynchronous JSON-RPC client in C to be embedded in the synchronous SIP server Kamailio using libevent, Unix Domain Sockets, shared-memory, multiprocessing, and DNS SRV.
  • Developed microservices using Twisted Python.
  • Wrote automated unit tests for the SIP routing infrastructure.
  • Designed and implemented a distributed, high-throughput, reliable call detail record (CDR) database system using NSQ, Couchbase, Riak, and Twisted Python. It allows for idempotent, real-time account balance tracking. Also implemented an integration test suite for this system.
  • Implemented the SMPP binary protocol (SMS) in Python.
  • Django development for the website.
  • Successfully advocated development best practices to the team, such as linting and code review.
  • Built from scratch a prototype for a new SIP server in Haskell.
DevOps Engineer @ Wetpaint Inc (2011 - 2013)
Social Media Newsite (Seattle, WA)
My job was to keep the websites and internal services running, test new releases, database management and migration, and release new versions of the site. I would also contribute to the frontend development whenever I had the opportunity.
  • Managed ~80 physical servers and ~10 AWS instances.
  • Introduced Puppet for all new and existing infrastructure.
  • Helped transition everything from SVN to Git.
  • Implemented an automated release process using Git.
  • Managed the Jenkins CI infrastructure.
  • Managed the F5 BIG-IP load balancer array.
  • Improved site response time for the slow Ruby on Rails CMS backend using Varnish and cache warming.
  • Centralized all server logging using syslog-ng.
  • PostgreSQL configuration optimizations.
  • Automated backup and replication for MySQL and PostgreSQL.
  • Frontend web development for main site. (HTML/ERB, CSS, JS/jQuery)
  • Designed and built a high-performance and distributed web analytics prototype using Couchbase and modified Nginx.
Systems Administrator @ General Electric Corporate - Enterprise Sensor Grid (2009 - 2010)
The Enterprise Sensor Grid is an intrusion detection system for GE that monitors their global internal network for malicious traffic. (Cincinnati, OH)
My role was to deploy, configure, monitor, and maintain the data collection hosts that monitored traffic at network endpoints scattered around the globe.
  • Deployed and managed ~150 physical servers running FreeBSD.
  • Introduced Puppet configuration management.
  • Replaced CVS with Git for some projects.
  • Built tools for automated deployments and management in Bash and Python.
  • Wrote a custom Splunk plugin to pull data from MS SQL Server.
  • Managed Nagios, BIND DNS, Apache, MySQL, WikiMedia, Wordpress, and Splunk.
Bachelor of Science in Electrical Engineering @ University of Washington (2008)
Embedded Computing Systems

Hobbies: Free and Open Source Programming, Rock Climbing, Biking, D&D, Arch Linux, Dad