Despite repeated calls to “Fail Forward Fast”, the uptake risk-taking for the sake of innovation has been slow. Here in Business Abstraction, we believe it must be attributed to misunderstanding of what Proof-of-Concept or PoC, the primary method of such innovation, should look like. We put together some recommendations on how to make a PoC a risk-free, no loss proposition. While the particular focus of these recommendations are on Australian Public Service, large Enterprises can find them useful as well.
Keeping it Small
A proper PoC should be easily affordable. The external costs should be kept under the procurement threshold. Also, a PoC should be limited in time, to limit the cost of public servants involvement.
While “fail forward fast” is repeated all the time, public servants are not prepared to admit failure of a larger effort. So a larger PoC can trigger a Sunk Costs Bias, and see the project pushed further and further in desperate attempt to get somewhere without changing much, in order to avoid political damage.
Target Validated Learning
The reason PoCs are needed is because there are promising ideas, however not enough knowledge to accept or reject them. Therefore the expected outcome from any PoC is knowledge that makes assessment of said ideas possible.
The watershed book Lean Startup introduced the concept of “Validated Learning” as the right kind of knowledge that should be acquired. It is not about general learning, like sending people to Scala, Python or “R” courses. Public Servants should learn the meaning of paradigms and technologies in the context of their problems. More about that in the last Chapter
Incorporate Strategic Business Analysis
The way an enterprise thinks of problems and desired solutions is mainly defined by legacy technologies, or lack of technologies – there are still many business processes that implicitly assume that files are made of cardboard. However an innovative project would not deliver significant improvement if forced to mimic outdated technologies.
To combat such limitation, a PoC should incorporate higher-level, Strategic Business Analysis that would approach senior executives to learn what should be achieved, without any assumption on how it should be achieved.
PoC is not a Product Evaluation
Evaluation of software platforms and COTS solutions is one of the most important areas of large enterprise IT, and some sites would call such evaluation a “Proof of Concept”. However in this paper we distinguish Proof of Concept from Product Evaluation:
- A PoC addresses a family of products and methods rather than a specific product. It seeks to establish expertise to evaluate products and/or methods within a specific category
- A Product Evaluation usually involves a crack team of engineers performing some magic with the system, then demonstrating the audience how the result would look. While definitely important at some stage, such demonstration does not teach the organisation, yet can only be properly assessed if the organisation is prepared for such demonstration
- A Proof of Concept should prepare an organisation for effective Product Evaluations through practical learning of the concepts and patterns the given family of products is based on
What absolutely cannot be trimmed from a PoC
That was addressed before, yet should be repeated. No PoC can compromise on the quaternity of:
- Needs understanding
- Knowledge transfer
- Concept Validation
While they look like four distinctive activities, in a PoC they are tightly intertwined:
- Needs understanding is critical to delivering the knowledge in the context of clients needs
- Knowledge transfer by the Consultant is the other side of Learning by the clients
- Concept validation can only happen on the basis of such learning.
Overall, such learning, both of Enterprise strategic needs in relation to new technologies, and of technologies in relation to the enterprise, is the very essence of a PoC.
Common project rules that a PoC can skip
The challenge of a successful PoC is running it with a budget and timeframe are way too small for any IT project with much less ambitious objectives. That makes a PoC impossible unless some limitations and activities of a traditional IT Project are not honoured
The following is an incomplete checklist of things you may be tempted to address, yet should not.
Coding unless specific to task and enterprise
the eventual solution would have a lot of code that would make the system relatively foolproof, easy to use etc. However, at the PoC stage, the question should be – what does it prove? For example, if it proves that a contractor employed for PoC can do nice GUI, that does not add any value.
Because of Internet companies and startups, for any task under the Sun there is a way to implement it in Petabyte scale. Say in Analytics, Apache Hadoop/Spark are rather ubiquitous, so are the experts, even if in demand. With sufficient funding any Agency can build a cluster onsite or in a cloud, hire a few specialists, and scale up processing beyond their needs. It is not a concept to be proved.
However a lot of Consultants want to add Spark/Scala to their resume on the way to more lucrative job in an Internet startup. It has to be up to the customer to scrutinise scalability-oriented tasks.
The same is applicable to transaction-oriented PoC, only instead of Hadoop/Scala, we would be talking about load balancing, Java, and many techniques designed for the purpose of scalability. Once again, hardly a PoC priority.
Say processing of a pilot dataset can take 40 seconds on a Spark cluster, or 4 hours on a Windows laptop running Vowpal Wabbit. Is a Spark cluster path a way to go?
It depends. You need two contractors over 3 months to setup the Spark cluster solution, and you can launch a necessary Vowpal Wabbit task from command line immediately, going with the Wabbit would save 3 months and massive costs. A Spark cluster setup would be needed eventually, however however not at the PoC stage.
Code support and continuity
Pretty much everything can be written in Java or C#, those are popular languages with skills easily available. Therefore any significant project that involves some coding is expected to employ one of those languages.
At the same time, it is nearly impossible to code anything in Java under the procurement threshold – a project would run out of money by the time they set up everything and write some boilerplate code.
Depending of the nature of the PoC, some exotic languages like Python, Groovy, Clojure, Prolog (in order of “exotic”) may increase productivity by order of magnitude or more. A Consultant engaged to conduct a PoC may be able to deliver required functionality in days. Sure, you would not be able to ding a pred-and-butter contractor capable of taking over such code, however it is not needed. If PoC is successful, the code would be re-written. If PoC is not successful, it is the learning of the public service that must be preserved, not the code.
There is a reason why “exotic” languages are shunned by the general IT public, yet loved by innovative consultants who push the limits of possible.
If the project does not deliver to production, they don’t do Operations, therefore don’t need DevOps.
When a project employs dozens of Developers, Analysts, DBAs, etc, proper processes are essential. When a one or two Consultants put together a quick PoC, you don’t need a DBA or a System Administrator policing who can create a Table.
Evaluating and learning technologies at the technical level
The objective of a PoC is to allow the client staff to evaluate the outcome of the proposed methods and technologies. It however should not include a Consultant engaged for the PoC learning about technologies and their advantages. If they don’t have pre-existing knowledge, they should not do the job.
Complex installations and integrations
As a general rule, installing open source software is a pain. At the same time, committing to a particular non-standard software platform from the start is not a very good idea, and may be a violation of probity rules for Public Service. However if there is a standard-compliant proprietary solution, that can be easily installed or installed by the vendor, it can deliver quicker PoC without providing advantage to the vendor.
This work is licensed under a Creative Commons Attribution 4.0 International License.