In a Small Company, NIH is Just Paranoia and Crackpot-ism

If you meet a math or science professor at a party and you don't feel like you have any intelligent questions to ask or small talk to make, just ask about his or her "crackpot file."  You're guaranteed amusing stories about sometimes smart, sometimes crazy, and always ill-trained "amateurs" who write in with easy proofs of hard things (like Fermat's Last Theorem) and hard proofs of impossible things (like squaring the circle).

In the software world, full-on crackpot code ideas tend to be self-correcting -- the code doesn't run, or doesn't produce the desired result. But crackpot-ism has a cousin in software. In large entities, this is referred to as NIH, or "Not Invented Here." An organization becomes so impressed with itself that it starts re-inventing more and more of the world, believing that its own in-house solutions are always better and more clever that what is in general use in the industry. NIH does not apply to "core" intellectual property, like a new algorithm for a search engine, where the whole point is to build something new or better; instead it applies to well-known components, like building one's own "better" database, when there is no evidence that an off-the-shelf database wouldn't work.

There are rare cases of NIH being beneficial, and beliefs that some organizations (the NSA and military) do indeed have more advanced solutions than what is generally available. But in most cases, and, in small software companies, in all cases, NIH is just crackpot-ism.

It is not uncommon for a startup to have a "genius hacker" doing a lot of the heavy lifting. Which is great. The problem comes when the genius hacker loses touch with the industry and the overall goal of the startup (make a product, make money) and begins creating his or her own "clever" and unusual architecture components. Common examples are inventing a new data storage engine (because none of the existing ones fit your application or are fast enough, and you don't understand the way these problems are normally solved and/or you haven't actually done a proper performance test anyway) and inventing your own TCP layer (because UDP is "more efficient" ... but -- wait! -- actually you need the features of TCP and surely you can figure this out better than the folks that developed it, they're so old and stupid anyway).

This flavor of crackpot-ism tends to corrupt the organization: as the system gets more and more peculiar, the company is increasingly dependent on the developer. New outside job candidates who figure out what's going on become increasingly reluctant to join the firm. The company lives in fear of the crackpot developer, because he insists that any substantial remediation would be foolish or have terrible schedule consequences.

Management may get an inkling of what's happening, but are more interested in the business benefit of achieving the next milestone than in the expensive, painful transition that will result from trying to bring some sanity to the product architecture.

And so it goes, with the strange architecture gaining more and more control over the company until the feasibility of products and features are determined by how well they fit into the crackpot's technology stack. Ultimately, the wacky implementation is driving the entire product roadmap, and the whole company is pwned.