What metrics matter when considering an OMS and what does “high performance” mean?

Technical performance: volumetry

Technical performance: speed

In our preceding post, we also mentioned the importance of the speed of an OMS. When considering the level of speed performance you need, you should take into consideration the tolerance of your customers waiting for information to be displayed,  or the reaction of a customer who orders an item that turns out to be out-of-stock (because the OMS was not updated fast enough), etc.

For instance, when addressing B2C customers, e-commerce platforms need to provide optimal response times to ensure the whole experience meets end consumer expectations. In fact, 40% of consumers will abandon a website that takes more than 3 seconds to load (Kissmetrics). 

Here are four metrics you can use to assess the basic ‘speed’ of an OMS:

• promise computing time – i.e., how long it takes to calculate and display the delivery options for a particular order

• order processing time – i.e., the time taken from order placement to order confirmation.
• inventory update time – i.e., how long it takes to update inventory levels after a transaction.
• monitoring delay  – i.e., how long it takes for “real-time” dashboards to monitor order execution KPIs to be updated .

A high-performance OMS will offer:

• a promise computing time of <1 second
• an order processing time of <1 second
• an inventory update time of <1 second
• a monitoring delay of <1 second 

Technical performance: reliability

Thirdly, and still under technical performance, you must consider reliability. When determining the reliability requirements for your OMS, you are to some extent measuring your organization’s risk appetite/aversion. For example, with many retailers doing the bulk of their business in the golden (fourth) quarter of the year, reliability issues then – should they occur – could have a major impact on revenue and profitability, as well as reputation and customer loyalty.

Here are three metrics you can use to assess the reliability of an OMS:

• system uptime (or system availability) – i.e., the percentage of time the system is operational and available.
• error rate – i.e., the number of errors encountered during order processing per thousand orders.
• mean time between failures (MTBF) – i.e., the average time between system failures

A high-performance OMS will offer:

• system uptime of >99.99%
• an error rate of <0.1%
• a MTBF of >1 year

Integration performance

As we said in our article on why you should care about OMS performance, your OMS is the beating heart of your retail operations – but, of course, no OMS operates in isolation. The need for your OMS to integrate with a wealth of other systems is simply a fact of life. With the benefits of an OMS highly dependent on effective integration with so many other systems, it is a critical metric to consider.

In this respect, it’s essential to consider carefully the third-party systems involved in the larger IT landscape around your OMS. OMS performance will be impacted by these other systems resulting in an ability to deliver (or not) frictionless end-to-end execution processes.

Here are two main ways to assess the integration capabilities of an OMS:

• API performance – i.e, the efficiency of APIs for integration with other systems.
• interoperability – i.e., the ability to seamlessly connect with different platforms and applications, including the number of available native APIS.

A high-performance OMS will offer:

• 99.9% of API calls with a response times of <200ms
• an extensive API library offering seamless integration with most third-party systems

Another metric to take into account is data synchronization time – i.e., the time taken to synchronize data between the OMS and other systems (ERP, CRM, WMS). However, there is no absolute answer for this, as the required speed differs depending on the system. For instance, stock and order synchronization with the WMS, ERP or in-store picking solutions need to be as close to “real time” as possible, while the CRM might only require a daily synchronization.

Business performance: scalability

An OMS is a major, multi-year investment. During this time, your organization almost certainly has objectives for growth. You will need to consider if/how your OMS will scale with the rest of your business.

Here are three metrics you can use to assess the scalability of an OMS:

• load testing – i.e., how the [whole eco]system will perform under increased load, e.g. doubling the number of orders processed.
• stress testing – i.e., evaluating how the system handles high-stress conditions and its breaking point.
• elasticity – i.e., the ability of the system to scale up or down in response to varying loads.

A high-performance OMS will:

• handle a 2x increase in load with <10% degradation in performance (load testing)
• operate effectively at up to 90% of maximum capacity (stress testing)
• scale up/down automatically within minutes (elasticity)

Business performance: user-friendliness

To reiterate again: your OMS lies at the core of your retail ecosystem. It must, therefore, be easy to use by all who must use it. There’s no point in having a top-notch OMS if it’s not adopted by users, and user-friendliness is a major driver for adoption.

Here are three metrics by which to assess its user-friendliness:

• user interface and user experience (UI/UX) design – i.e, the intuitiveness and simplicity of the platform.
• training time – i.e., the time required to train employees to proficiently use the system.
• user feedback – i.e., collect and analyse feedback from users regarding their experience.

A high-performance OMS should offer:

• a highly intuitive, easy-to-navigate interface (for instance, fewer than 1 pain point per 10 pages - based on Nielsen's Heuristics, which provide a set of rules where any violation can be considered a pain point).
• training time of <1 week per individual user
• high satisfaction scores, with minimal complaints

Business performance: flexibility

In our preceding article, we highlighted the importance of an OMS’s flexibility to adapt to meet the ever-changing needs and circumstances of a customer’s business. For example, its ability to:

• react to new customer expectations, or to enable new customer behaviors;
• to enable new business processes and the creation of new workflows; and
• to meet the external challenge or responding to increased competition

It might appear difficult to assign metrics to such attributes. However, it is not impossible - if you consider these attributes of an OMS to be the outcome of possessing other attributes that we have discussed. For example:

• an OMS that is high-performance in terms of speed is likely to meet the speed expectations of customers.
• an OMS that is high-performance in terms of integration is likely to be best-placed to enable new customer behaviors, new business processes and new workflows
• an OMS that is built to MACH principles is likely to be best-placed to always use the best, most up-to-date micro-services, enabling it to always respond to external challenges and/or competition.

Nonetheless, flexibility is a key attribute of the overall performance of an OMS, which you will want to explore with your OMS provider.

Where to find performance information about an OMS

Of course, knowing the metrics associated with OMS performance is one thing. Obtaining these metrics, in order to make comparisons, is another. But there are sources. Try any or all of these:

• vendor documentation and demos: most vendors provide detailed documentation and live demos of their OMS. These can give insights into their system’s speed, reliability, and user interface.
• case studies and testimonials: reviewing case studies and testimonials from other businesses can provide real-world performance data. Look for case studies relevant to your industry or use case for the most accurate comparison.
• third-party reviews and comparisons: G2, Capterra, IHL, Gartner Peer Insights and others offer analyses of various OMS solutions, often including performance metrics and user experience assessments.
• technical specifications: vendors usually publish technical specifications, including API response times, uptime guarantees, and scalability features. This information can be critical in assessing performance.
• pilot programs: it is tricky with a system as fundamental as an OMS, but pilot programs or trial periods may be possible, enabling you to test the system’s performance in your own environment.
• industry forums and user groups: engaging with these may provide insights into the performance of different OMS solutions from actual users, along with unfiltered feedback and practical advice.

By understanding the key attributes of performance and what signifies high performance, you can make an informed decision that supports your business needs. Keep in mind that the whole point is to see where you need your OMS to perform best, to best support you in your business goals and ambitions; and recognise the importance of balance: for instance, there’s no point in having the capacity to process 10 order per second, if the inventory update time is 2 seconds. 

See how Kbrw’s high-performance compares to the competition, as measured by this IHL Order Management Market study.