POC vs. POV: Evaluating ROI with fraud protection vendors

Plenty of anti-fraud vendors out there will tell merchants they understand the need to protect and partner and will deliver on that promise. But as a prospective buyer, how can you be sure that you will get the best possible return on your investment?

Enter the proof of concept exercise — or more accurately, proof of value exercise (more on that distinction later). Proof of concept (POC) and proof of value (POV) exercises are fast becoming the industry standard in terms of anti-fraud vendor selection since they provide merchants an opportunity to observe firsthand how a provider would impact their traffic. This allows merchants to assess an anti-fraud vendor’s effectiveness and accuracy within their own operational environments before making a full commitment. 

However, there are many important factors to consider before engaging in a POC or POV exercise. As a technical implementations leader in the anti-fraud industry who has overseen dozens of POV exercises in a variety of formats, I’d like to share my insights on how merchants can design effective product trials to ensure that they’re getting the most value out of this important decision.

Proof of value vs. proof of concept

Before diving deeper, we need a clarification of key terminology. While “proof of concept” (POC) often describes preliminary “try before you buy” exercises in the antifraud industry, this term falls short of the true objective.

A “proof of concept” (POC) demonstrates a product’s basic functionality in a controlled environment, focusing on feasibility. But a “proof of value” (POV) extends this by showcasing real-world, quantifiable value in the context of the merchant’s actual operations by measuring impact on revenue, conversion rates and user experience.

Therefore, when working with an anti-fraud vendor, it’s crucial to focus on the real-world value delivered, not just feasibility. At Signifyd we ensure that we remain focused on the outcome by using the term proof of value, or POV, to describe these exercises. 

Types of POVs

There are two major dimensions that distinguish POVs:

• Integration method
• Number of prospective vendors considered

For the first dimension, there are generally two options: non-integrated (also called “flat file,” “offline” or “historical” POVs) and integrated (also referred to as “online” POVs). A non-integrated POV involves analyzing past transaction data without real-time integration into the merchant’s production system. Merchants typically share transaction data and evaluation results in one-time batches via CSV files. During integrated POVs, the anti-fraud vendor fully integrates into the merchant’s live system, analyzing actual transaction data in real time to assess performance.

Integrated POVs break down into those that apply decisions from a prospective vendor or vendors (called a “split decisioning” or “live” POV) or simply record the responses for analysis without applying them in production, which we call a “shadow mode” POV.  Merchants almost never apply decisions from non-integrated POVs to production traffic, so the classification as a type of “shadow mode” POV is generally redundant.

As for the second dimension, merchants are almost always considering the performance of a prospective vendor against their incumbent anti-fraud provider. The question is whether they are considering the performance of multiple providers simultaneously (in what we call a “competitive” POV) or a single provider (in what we call an “incumbent comparison” POV). 

These dimensions intersect with one another, leading ultimately to the following types of POVs:

• Integrated (“online”)
  • Shadow mode
    • Competitive
    • Incumbent Comparison
  • Split decisioning (“live”)
    • Competitive
    • Incumbent Comparison
• Non-integrated (“flat file” or “offline”)
  • Competitive
  • Incumbent comparison

Considerations for each type of POV

With so many different types of POVs to choose from, it is essential for merchants to carefully evaluate their specific needs and constraints before deciding on the best approach. There are several key considerations for each type of POV.

Applicability of results

Anti-fraud vendor performance is highly contingent on the quality and completeness of the data received. Certain POV modalities such as flat-file POVs generally mean key data points aren’t available, such as device fingerprint. That can lead to inconclusive results.

The absence of device fingerprint and other data points will inevitably impact vendor performance, meaning merchants cannot assume POV results will directly translate to real-world performance. Furthermore, as fraud trends change over time, merchants cannot necessarily extrapolate past behavior and results to predict future performance accurately.

If real-world performance applicability is of paramount importance, then an Integrated POV that captures all production data points and real-time decisioning may be the right choice.

Integration complexity

Live POVs tend to offer the most comparable results to a full integration but they typically require more complex integration efforts (even more so when they are competitive, multi-vendor exercises). Merchants should ensure they have the necessary technical resources and support from the vendor to implement and monitor these trials effectively.

If developer resources are scarce, a flat-file POV can be a viable path forward, though results can often be more directional than decisive. Note however that non-integrated POVs come with their own technical requirements.  Offline POVs can involve large data scales, often requiring significant data engineering to collect all the necessary data for an effective evaluation.

Decision validation impasse

There is an inherent challenge in anti-fraud POVs when evaluating the accuracy of competing fraud decisions. When a competitor claims they would approve a transaction that the incumbent rejects, it’s impossible to verify who is correct without knowing the true outcome. We refer to this uncertainty in evaluating the effectiveness of prospective vendors as “decision validation impasse.”

If the uncertainty presented by decision validation impasse is too problematic for your evaluation methodology, then an integrated split decisioning POV that enables multiple vendors to decide on their own traffic may be the right decision.

Figure: Decision validation impasse

Interpretability

Given the inherent challenge presented by the decision validation impasse as well as the skewed and cyclical nature of fraud attacks, interpreting POV results and inferring real-world performance can be challenging. This challenge becomes exponentially more complex as merchants introduce additional vendors to any POV exercise.

In general, we highly recommend limiting the number of vendors considered in a POV to one or at most two competing vendors beyond the incumbent. Otherwise, the uncertainties introduced by decision validation limitations and the random distribution of fraud attempts will muddy results beyond interpretation.

Time frame

Merchants should always consider the time frame available for making a decision. Integrated POVs tend to require more time up-front on the integration as well as an extended integration period, whereas flat-file POVs can offer a more rapid evaluation route based on historical (rather than live) data.

Although flat-file POVs can offer quicker results, there’s no escaping the eventual need for system integration. Be sure to consider the entire lifecycle of vendor selection (including onboarding) in your POV structure. If development resources are likely to become scarcer in the future, you may prefer to accelerate the integration and perform an Integrated POV, ensuring your vendor is already connected to your system if you decide to move forward with them.

Defining success criteria

Independent of structure, it is crucial to define clear success criteria to evaluate vendor performance during a POV. The nature of the POV structure may influence those criteria, but in general, merchants evaluate POVs based on a vendor’s ability to demonstrate:

• Increases in approval rate
• Decreases in fraud or chargeback rate
• Decreases in false positives (rejections of non-fraudulent orders)
• Reduction in operating costs
• Pricing

A methodology that we’ve seen increasingly adopted in the industry is combining all of the above metrics in a single “total cost of fraud” (TCOF). While this has the advantage of unifying all factors into a single, comparable number, it does carry the risk of obfuscating the impact on individually important criteria such as approval rate or false positive rate, especially during critical point-in-time analysis (e.g. examining how a vendor performed during a flash sale or a fraud attack). 

Regardless of methodology, defining clear and quantifiable success criteria is crucial for arriving at an objective, data-driven decision when selecting an anti-fraud vendor. 

Alternatives to POVs

While POVs are a robust method for evaluating anti-fraud vendors, they come with drawbacks such as ambiguity of results, high effort of execution and longer timelines in the vendor selection process, among others. Merchants seeking to streamline their decision-making process may consider alternative strategies. 

One alternative to a POV is a “declines-only” integration, where a prospective vendor analyzes only the transactions that the incumbent anti-fraud provider is rejecting. This methodology is particularly attractive if the prospective vendor offers a chargeback guarantee for the orders it approves.

Since merchants would not fulfill declined orders, any incremental approvals by the prospective vendor represent risk-free additional revenue that would otherwise be left on the table. Although performance is limited to a smaller segment of overall traffic, declines-only integrations let merchants become familiar with a new provider’s tools, operating style and accuracy.

Another alternative is a limited-term pilot engagement with a 90-day assessment window. This pilot program allows merchants to engage with a vendor on a short-term basis, providing a real-world integrated testing environment without the pressure of a long-term commitment or the evaluation complexity of a POV. The 90-day evaluation period enables merchants to assess the vendor’s impact on fraud detection and approval rates as well as their working relationship, while retaining the flexibility to discontinue the partnership if the results do not meet expectations.

Achieving comprehensive fraud protection and business success

Ultimately, when selecting a fraud vendor, merchants should seek a partner who provides confidence that they will not only mitigate fraud risks but also support and enhance overall business performance, driving growth and customer satisfaction.

POV exercises offer an opportunity to see an anti-fraud vendor in action, not only in terms of numerical performance, but also in terms of understanding how they will partner with you and your team in the complex, nuanced, data-centric conversations that are required to execute a successful POV and ultimately a long-term partnership. 

At Signifyd, we believe transparency, objectivity, and partnership should rule the day when selecting an anti-fraud vendor. We strive to earn our customers’ trust on those fronts in any POV we engage in. With the right approach, merchants can make an informed and confident decision that ensures long-term success and resilience against fraud.

Interested in learning more about POVs? Let’s talk.