The Palantir Developer Reformation
How to Solve Palantir’s Developer Problem
Disclaimer: This article is my own opinion and does not reflect the views of Palantir or its partners.
Palantir’s Forward Deployed Engineering (FDE) model has proven to be a powerful tool for product development in high-stakes environments where victory is the only possible outcome. Palantir CTO Shyam Sankar has often said, “If the problem could have been solved with a requirements document, it would have.” Under this model, Palantir embeds a team tasked with solving their customer’s most challenging problems. In the military context, this can mean working with the soldiers tasked with carrying out a mission and famously helicoptering into active warzones to ship software updates. In the commercial context, teams will colocate on-site and unpack the customer’s business and their most challenging problems. Solutions are then rapidly put together using Foundry’s existing Lego blocks, and where gaps emerge, the FDE works with PD teams to rapidly build and ship new core technology. A debriefing also takes place with PD teams post-deployment to gather lessons learned and inform the product roadmap. This product development method tightly couples the FDE and the PD to ensure the shape of Palantir products matches the shape of the problems they are attempting to solve (the world’s most challenging).
While the FDE model has ensured Palantir’s products avoid the rapid rate of enshitification infecting the majority of SaaS companies, it does exclude external software engineers from interacting with the PD teams. This has led to a developer experience (DevEx) that often leaves the developer feeling ill upon first exposure to the platform. The “medicine” for most developers is to think and act more like FDEs and less like software engineers. However, this leaves the engineer underutilized and unable to support a larger segment of their organization in the same way PDs support FDEs. This includes incorporating observability tools that help detect and monitor errors and performance issues, building new core modules to extend Foundry’s capabilities, and automation tools that speed up solutions and automate quality. But for Palantir, supporting these developers can be a real challenge.
As an example, let’s consider the lack of a proper command line interface (CLI) for Foundry. CLIs enable developers to automate common tasks within a platform. All Hyperscalers have robust CLIs that are used to orchestrate services, saving huge amounts of time that would otherwise be spent navigating their consoles and engaging in ClickOps (as opposed to DevOps). Beyond being a huge time saver, scripts can automate quality by ensuring projects have the appropriate guard rails and structure across all teams. This is a near-impossible task without automation. So why does Foundry not have a CLI? A better question, given the FDE model, is why would it? The category of Foundry users (in sufficient quantity) that would use a CLI did not exist. In addition, many parts of Foundry are not ready to be exposed via CLI either because they lack a public interface or appropriate documentation to consume one. This is precisely because of Palantir’s focus on the mission, manifested in the FDE model, which prioritized victory, not self-pleasuring technical philosophy. A CLI is a solution in search of a problem when viewed through this lens.
Culturally, PD teams are not equipped to empathize with developer problems and will attempt to reframe them as a Foundry-shaped solution. For example, rather than building a CLI, a PD team might ask, “What do you want to use a CLI for? What if we build a Foundry module to help you build templates, create and assign groups faster, etc.?” PD teams have been trained to find the value and abstract the delivery mechanism. The essence of what Palantir’s PD teams will have to learn can be summarized as follows:
The more of your stack an external software engineer can access, the more value they can create on your behalf. While you must control the degrees of freedom the developer has to effect change in your core stack, you should not restrict their ability to be additive at every level of the stack. This is the equivalent of a force multiplier for your business and an essential key to unlocking growth in the face of increasing costs that are proportional to the customer count.
For Palantir, this means adding a new dimension to product development that can empower external developers. Below are some practical examples of these dimensions.
Foundry Dev Tools (FDT)
FDT was created by Merck KGaA, Darmstadt, Germany (MKDG), and provides a CLI for Foundry and local caching to speed up the development process for data transformations. MKDG has a large-scale Foundry deployment that, without these tools, may have been impossible.
Foundry Compute Module
Compute Modules allow developers to create their own containerized workloads with Docker. Exposed functions are published to Foundry, consumable in applications like Workshop and callable inside other functions.
Bring Your Own Container (BYOC)
BYOC allows developers to customize their environment for data transformations, leveraging their preferred data processing technology.
A methodology for identifying such solutions is less clear. One approach might be to think of low-level solutions that solve a category of Foundry problems. For example, allowing the developers to run arbitrary compute with Foundry Compute Modules solves the entire category of problems with Foundry Functions related to the custom runtime, including the supported ECMA script version, missing node APIs, etc. Future versions of Functions could allow developers to use images published to the container registry as their function’s runtime. Another example might be sandboxing log streams at the customer level for ingestion into common solutions like Sentry. These log streams could power a Foundry module for error monitoring and troubleshooting while unblocking customers looking to improve Foundry observability and availability (or compliance).
If Palantir can extend its product development methodology to include external software developers, it will have solved one of its toughest challenges and proven (once again) that it can reinvent itself. So far, all signs point to a positive outcome. Palantir recently launched a community forum to support Foundry developers and offer free developer stacks. They are also busy improving documentation and education resources. Its inaugural BuildCon is taking place this November. Partners also host regular developer meetups (I am hosting one in Denver this October). The ability to extend Workshop is reportedly in the works, empowering React developers. Palantir also connects PD teams with their customers’ external engineers more frequently and has created channels to deliver developer feedback. While many challenges remain, Palantir is on the path to widespread developer adoption and all the benefits that come with it.
