Trying to be cogent.
I’m a software engineer with a focus on designing for simplicity, reliability and stability in distributed systems. At SoundCloud I’m the Tech Lead of the Data Platform, working on the architecture of our data infrastructure, and guiding how we work with data infrastructure across the company.
I cycle a lot, and my big event of the year has been the Haute Route Alps, riding in the mountains for a week.
The next exciting change is the new tiny human about to join us!
Google's 2015 paper on the Dataflow model describes general solutions to general data pipeline processing problems. The terms they use have been helpful to me in understanding patterns in these problems.
We have a good solution for ownership of services in a microservices architecture. We can learn from this to define ownership of datasets in a way that reduces the total cost of maintenance and integration across teams.
A blockchain allows independent parties to make verifiable statements. This works with bitcoin, whose value comes from the system itself, but fails in applications where the value is external.
Two-phase commit is a long-established means of keeping two resources strongly synchronised. These days it's not so sexy, but it's an important piece of heritage of distributed computing.
Record shredding allows nested data structures to be considered in a sort-of-tabular way, and stored in a columnar data store. This post describes the intuition behind how this can be done preserving message structure, from Dremel and Parquet.
An incredible and joyful (and exhausting) week in beautiful places. What an experience.
One last push, a single time section of 100km to get through. I caught and passed plenty, and kept my top 100 GC placing – I finished the Haute Route Alps!
Yesterday's ride really took its toll, but once the timing started today the legs woke up again, and felt fresh and strong for the last day of the High Alps.
The queen stage of the event, long and high, and on a really hot day. The first half went well; the second half I pushed too hard, paid for that on the last climb and in recovery that evening.
The classic mountain time trial. I pushed a little harder than planned, maintained overall position, and spent the afternoon eating.