In the 1960s, Bank of America and IBM built one of the first credit card transaction processing systems. Although those early mainframes processed just a fraction of the data compared to that of Amazon or Capital One, the engineering was complex for the day. Once credit cards became popular, processing systems had to be built to handle the load and, more importantly, handle the growth without constant re-engineering.
In the 1970s, IBM’s monopoly was curtailed enough for other startups such as Amdahl, Digital Equipment Corporation, and Oracle to emerge and begin providing IBM customers with alternatives. DEC built the VAX that provided superior price/performance to IBM mainframes, but without compatibility. Amdahl (whose namesake, Gene, designed the IBM 390) provided a compatible alternative that was cheaper than an IBM mainframe. Companies could develop and sell their own products or services and thrive in the post-monopoly world.
These pockets of alternative value eventually led to silos of vendors and silos of expertise within IT organizations that were aligned with the vendors. Like Amdahl, Oracle directly benefited from technology that IBM developed but never productized. Larry Ellison’s genius was to take IBM’s relational database technology, System R, and place it on the VAX creating one of the first enterprise software companies in the post-mainframe era.
When products within silos or niches were sold to customers, putting the system together was no longer any single vendor’s responsibility; it became their customers' jobs. Today there are so many vendors for every imaginable silo—network switches, storage switches, storage arrays, servers, operating systems, databases, language compilers, applications—and all the complication and cost that comes with the responsibility.
Big systems integrators like Accenture and GDIT attempt to fill this gap, but they also operate within the constraints of IT departments and the same organizational silos established by vendors. Silos are the price paid for the post-mainframe alternatives to IBM. Silos obfuscate the true nature of computing platforms as homogeneous systems of interconnected hardware and software.
One of the surprises awaiting enterprises is that webscale technologies like Hadoop and Kafka are DIY. Whatever cluster they stand up, it comes from the factory without applications or data. In order to populate the cluster, data must be emancipated from their own technical and organizational silos. Legacy silos— whether they’re infrastructure, organizational, or vendor-sponsored—must be replaced with a platform approach.
In the past, organizations and agencies were satisfied with purchasing a SQL prompt and building their own applications. Today, those groups don’t want to read raw data; they need to render it to make better business decisions. Implemented successfully, modern infrastructure delivers the data to the right place (the analytics layer) at the right time for the right cost. If the infrastructure can aggregate a richer set of data such as tweets, videos, PDFs, JPGs, and SQL, then the analytics layer has a better chance of delivering actionable intelligence for the business.