The AI/ML market is teeming with solutions, each promising revolutionary outcomes. For businesses, cutting through the hype to identify genuinely valuable tools can feel like navigating a minefield. How do you discern actual capabilities from puffery? This article offers guidance on making an informed decision when selecting an AI/ML vendor.

Understanding the Hype

In the world of marketing, it’s not uncommon for products to be presented in the most dazzling light possible. With AI/ML, the “dazzle” often comes in the form of:

1. Broad claims about the product’s applicability to any industry or problem.
2. Technical jargon used to confuse rather than clarify.
3. Unsubstantiated promises of high accuracy or performance.

Questions to Guide Your Selection

Solution Specificity

If all you have is a hammer, you will see everything as a nail. In this case, your vendor should be able to recognize the problem you are trying to solve and if the solution is a good fit for you. If they insist everyone can use their technology, they don’t understand their solution well.

• What specific problem does your AI solution address?
• How is this solution better or different from other AI products on the market?

Good Response: A focused answer that highlights specific industries, use-cases, or challenges.

Bad Response: Vague, catch-all answers implying universal applicability.

Technical Details

These questions can help you determine if you are being sold snake oil or actual AI/ML solution. I would expect the vendor to be able to explain their solution in terms you can understand and identify the type of AI being used. In addition, the vendor should be able to go deep with your technical people.

• What algorithms or techniques does the solution use?
• Can the model be retrained or fine-tuned for our specific dataset or use-case?
• How does the model handle anomalies or outliers in the data?
• What are the hardware and software requirements for the solution?
• Is the solution Cloud Native?

Good Response: Clear, jargon-free explanations showcasing transparency.

Bad Response: Evasive answers or a barrage of technical terms without context.

Data Requirements

This is one of the areas that will likely require Extract Transform and Load (ETL) of your data into the new system. This will be time consuming, and I would expect a well defined process, robust support, and robust/custom tooling from the vendor.

• What type and amount of data are required to effectively use the solution?
• How does the solution handle data quality issues?
• Is there any data preprocessing or cleaning required before input?

Good Response: Detailed specifications about data quantity, quality, and format, demonstrating the vendor’s understanding of data intricacies.

Bad Response: Overly simplistic answers ignoring data intricacies.

Performance Metrics

There will be standard metrics available as well as new metrics that you are likely unfamiliar with. Reporting on the solution should be available. Accuracy of response should be considered for any generative AI/ML or chatbot solution.

• What metrics do you use to measure the solution’s performance?
• Can you provide benchmarks or case studies demonstrating its effectiveness?
• How does the solution perform in real-world, non-ideal conditions?

Good Response: Actual numbers, preferably compared against industry standards or competitors, along with real-world success stories or case studies.

Bad Response: Ambiguous statistics without context or reference points.

Security and Privacy

Regulations are starting to show up for AI/ML solutions. While there will be some reuse of existing regulation like HIPAA, I would make sure the vendor has insight to the regulatory and privacy concerns of your industry. Output of the systems should be protected from data leaking as well.

• How does the solution ensure data privacy and security?
• Are there any compliance standards or certifications (like GDPR, HIPAA) that the solution adheres to?

Good Response: Mention of recognized compliance standards (GDPR, HIPAA, etc.), data encryption methods, and robust policies for data handling and storage.

Bad Response: Generic assurances without tangible measures.

Scalability and Integration

Scalability should be able to be demonstrated. Integration may be difficult here based on how old the systems you are asking to be integrated with. Most vendors at this stage are probably more focused on their underlying AI/ML technology and not their integration capabilities.

• How scalable is the solution? Can it handle growing amounts of data or more complex tasks as our needs evolve?
• How easily can the solution integrate with our existing systems and infrastructure?

Good Response: Evidence of successful integrations with diverse systems, and scalability proofs from past clients or projects.

Bad Response: Broad statements lacking specificity or past evidence.

Ethical Considerations

If you are using an AI/ML solution I would be very cautious of any vendor who does not have good answers to these concerns. This is a major issue for black box AI/ML solutions which are difficult to determine how they are making their decisions.

• How does the solution account for biases in the data?
• Have there been any third-party audits for ethical considerations?

Good Response: Honest acknowledgment of AI/ML limitations and steps taken to minimize biases, perhaps through diverse training data or third-party audits.

Bad Response: Dismissal of potential issues or oversimplifying complex ethical concerns.

Customization and Flexibility

The power of AI /ML solutions is the ability to ingest your data and give you very specific tailored solutions. I would expect a solution to be very customizable.

• If possible, how can the solution be customized to better fit our specific needs or industry? How long will the process take?
• How flexible is the solution in adapting to changes or new requirements in the future?

Good Response: Outlining the steps for customization, along with indicating that the system was designed for customization with techniques such as modularization.

Bad Response: Vagueness of response, or lack of knowledge of a customization process.

Training and Onboarding

With many of the solutions being new, or relatively new, I would expect training to be on the weaker side, with more of an emphasis on customer success as demonstrated through resources from the company with your from start to finish.

• Do you offer training or onboarding sessions for our team?
• Are there user manuals, documentation, or tutorials available?

Good Response: Vendor supplied training, strong user community, hands on vendor supplied training along with onboarding. A vendor willing to commit to your success.

Bad Response: Commitment to help but no offer of an onboarding process or manager to help with you success.

Maintenance and Support

I would expect highly available support from the vendor on an AI solution. This is relatively new technology and you do not want to be alone in troubleshooting.

• What kind of post-purchase support do you offer?
• How frequently are model updates or improvements released?
• In case of issues, what is the expected turnaround time for support?

Good Response: Comprehensive support provisions, regular updates, training sessions, and a dedicated contact point for troubleshooting.

Bad Response: Vague promises without a structured plan.

User Feedback and References

If you are first out of the gate with a solution, there is a high level of risk associated with purchase. Ensure you have references from current customers/clients of the company. You don’t want to be paying for R&D of the vendor on your dime.

• Can you provide references or testimonials from other clients who’ve used the solution?
• How has the solution been received by users in similar industries or with similar needs?

Good Response: Actual names or companies (with their permission) that you can contact for a more unfiltered perspective.

Bad Response: Reluctance or diversion from the request.

Licensing and Costs

AI/ML vendors are probably going to throw different types of licensing than you are used to for their solution. Make sure you understand all the nuances. measures, and examples of the costs so you don’t end up with something you cant afford.

• How is the solution priced? Are there additional costs for updates or support?
• Are there costs associated with retraining?
• What is the licensing model? Are there restrictions on usage or distribution?

Good Response: Transparent breakdown of all costs, including licensing, updates, and any other potential fees.

Bad Response: Ambiguous pricing with potential hidden fees.

Validating Vendor Claims

In addition to asking the right questions, consider:

• Requesting a Demo and/or Proof of Concept: This provides firsthand experience with the solution.
• Consulting Independent Reviews: Neutral platforms or industry analysts can offer objective insights.
• Engaging with your Network: Personal recommendations or experiences from peers can be invaluable.

In the evolving AI/ML landscape, due diligence is paramount. As vendors jostle for attention in a crowded market, businesses need to equip themselves with the right questions and a critical mindset. The right solution will not just have impressive capabilities on paper but will demonstrate genuine value, align with your specific needs, and come from a vendor that views your success as their own.