Buying an AI tool can feel like buying a magic trick. The demo looks great, the sales deck is polished, and then you realise the real questions are boring ones about data, accuracy, and what happens if you want out.

That is normal. AI tools can genuinely help, but you only get the benefit if you stay in control of your information and your options.

Start with one simple outcome

Before you ask a vendor anything technical, get clear on what “success” looks like for your business. Otherwise you will end up comparing features instead of results.

A good starting point is: what decision or task will this tool improve, and how will you measure that improvement?

  • A single use case. “Summarise support tickets into a weekly trend report” beats “make us more efficient.”
  • A baseline. How long does it take today, what does it cost, what is the error rate?
  • A human owner. Who in your business is responsible for the output being correct and appropriate?

Once you have that, the vendor questions get much easier.

Data handling: where does your information go?

Most AI risk for growing businesses is not “the model becomes evil.” It is much more practical: sensitive data ends up somewhere you did not expect, or more people can access it than you realised.

Use these questions to force clarity.

  • “What data do you collect from us, and what counts as ‘our data’?” Ask them to list it: prompts, uploaded files, chat history, usage logs, account details, and any data pulled from connected systems.
  • “Is our data used to train your models, or improve the product for other customers?” You want a plain yes or no, plus what is excluded (for example, “not used for training” but used for human review).
  • “Who can see our data inside your company?” Ask about role-based access, support access, and whether engineers can access customer content by default.
  • “How long do you keep prompts, files, and logs?” Look for a specific retention period, not “as long as necessary.” Also ask what happens when you delete content.
  • “Where is the data stored and processed?” This matters for compliance and customer contracts. If you have Texas and Florida operations plus customers in other states, clarity on location and residency still helps.
  • “How is data protected in transit and at rest?” You are listening for basics like encryption, and for whether you can control keys in higher-security scenarios.
  • “What happens if we connect it to Microsoft 365, Google Workspace, or our CRM?” The risk often comes from the connectors. Ask what permissions it needs, whether it respects existing access controls, and how it prevents oversharing.
  • “How do you handle prompt injection and data exfiltration risks?” You do not need to be a security expert, but the vendor should have a coherent answer and documented mitigations.

If the vendor cannot answer these clearly, assume they have not done the work.

Accuracy: how wrong can it be, and how will we know?

AI output often sounds confident even when it is incorrect. That is not a moral failure, it is a known behaviour of generative systems. Your job is to make sure incorrect output does not quietly become a business process.

Ask questions that turn “it is smart” into something testable.

  • “What does the tool do when it does not know?” You want to hear about guardrails, refusals, and how the system avoids inventing details.
  • “How do you measure quality for our use case?” A vendor should talk about evaluation, not just anecdotes. If they claim accuracy, ask how they tested it and what “accuracy” means in practice.
  • “Can it show its sources, and can we restrict sources?” For internal knowledge tools, you want the ability to limit answers to approved company content, not the open web.
  • “What is the human review workflow?” For anything customer-facing, financial, legal, or HR-related, ask how outputs are reviewed, approved, and logged.
  • “What are the known failure modes?” A trustworthy vendor will tell you where the tool is weak (for example, edge cases, ambiguous requests, or rapidly changing information).
  • “How do you handle drift over time?” Models, data, and business processes change. Ask how performance is monitored after launch, not just during onboarding.

A good vendor will not promise perfection. They will show you how they reduce errors and how you can catch them.

Lock-in: how hard is it to leave later?

Lock-in is not only about pricing. It is about your data, your workflows, and your team learning a tool that becomes difficult to replace.

Get answers now, while you still have negotiating power.

  • “What data can we export, in what format, and how often?” Ask for specifics: conversation history, embeddings, custom instructions, settings, audit logs, and any fine-tuning or configuration.
  • “If we cancel, what happens to our data and when is it deleted?” You want a clear offboarding process and timeline.
  • “Do you rely on a single underlying model provider?” Many vendors sit on top of larger platforms. That is not automatically bad, but it affects continuity and pricing.
  • “What is your plan if a model or feature is discontinued?” This is where smaller vendors sometimes struggle. The answer should include a migration path.
  • “Can we run this in our tenant or our cloud?” For some businesses, keeping the system inside your existing environment reduces both risk and lock-in.
  • “What is the minimum contract term, and what changes pricing?” Ask what happens when usage grows, when you add integrations, or when you need higher data protection.

If the tool becomes central to operations, you want an exit plan that is annoying, not catastrophic.

Security and governance: the grown-up questions

Even if the tool is “just for productivity,” it still touches identities, permissions, and business data. Treat it like any other vendor that could affect operations.

  • “Do you support SSO and MFA, and can we enforce them?” If you cannot control access centrally, you will regret it.
  • “What audit logs do we get?” You want visibility into who used the tool, what data it accessed, and what actions it took.
  • “What is your incident response process?” Ask how you will be notified, in what timeframe, and what information you will receive.
  • “Do you have independent security assessments?” Look for third-party testing, certifications, or at least a clear security programme.
  • “Can you complete a vendor security questionnaire?” If they cannot handle a standard vendor review, that is a signal.

CISA publishes vendor supply chain risk management templates for a reason. Mature vendors expect this.

A quick way to use this checklist

If you want a simple process that does not drag on for months:

  • Round 1 (30 minutes). Ask the data-handling questions. If the answers are vague, stop.
  • Round 2 (60 minutes). Ask accuracy questions using your real use case and a small set of real examples.
  • Round 3 (contract review). Confirm retention, training use, export, deletion, and incident notification in writing.

Want a second set of eyes?

AI tools are moving fast, but the fundamentals are steady: control your data, verify outputs, and keep your options open. If you would like help reviewing an AI vendor, running a risk check, or setting guardrails for safe use, the Flexnet Networks team can help you do it.

Sources