Skip to main content

AI Assessment & Audits

Basic Advanced

Service: AI Assessment & Audits · Typical shape: 1–3 weeks, fixed scope; 1 lead engineer + 1 data/ML engineer, with a security or compliance specialist for audit-heavy scopes

What we deliver​

A decision-ready picture of where the client stands and one concrete first use case with the data it needs, mapped to the regulatory obligations that apply.

DeliverableContent
Use-case portfolioLong list, scored shortlist, recommended first use case with success metric
Data readiness reportPer data source: owner, access, quality, sensitivity, legal basis, gaps, and remediation steps tied to the chosen use case
Governance gap analysisCurrent practice vs ISO/IEC 42001 controls, NIST AI RMF functions, EU AI Act role and risk class, GDPR/DPIA needs
Architecture and security reviewTarget architecture sketch; OWASP LLM Top 10 2025 and MITRE ATLAS findings for existing AI systems
Roadmap90-day plan: PoC scope, data fixes, governance actions, owners
Executive readout10–15 slides plus the full report

The output flows directly into a GenAI PoC, and for cloud-side findings into a Cloud & Migration Assessment.

Assessment method​

Week1-week variant3-week variant
1Kickoff, 5–6 interviews, data source inventory, long list, scoring workshopKickoff, 8–12 interviews, data inventory, document review
2—Data profiling on samples, scoring workshop, governance mapping, security review of existing AI systems
3Readout at end of week 1 (light report)Target architecture, DPIA screening, roadmap, readout

Day-0 document request​

  • Org chart for IT, data, security, legal/DPO, and the business units in scope
  • Inventory of existing AI/ML systems and AI tools in use (including SaaS copilots)
  • Data catalogue or list of core systems (ERP, CRM, DMS, ticketing, data warehouse)
  • Information security policies, data classification scheme, records of processing (GDPR Art. 30)
  • Existing DPIAs, vendor contracts and DPAs for AI services
  • Cloud and infrastructure overview, including GPU availability

Implementation steps​

1. Interviews​

Run 45-minute interviews with two people from Lieber (one leads, one takes notes into the template). Group by role.

Executive sponsor

  1. Which business outcome would justify AI investment this year, and how is it measured today?
  2. What has been tried with AI so far, and what was learned?
  3. What is the risk appetite for data leaving the company or the EU?
  4. Who decides on go/no-go for a PoC and for production?

Business process owner

  1. Walk us through the process step by step. Where do people spend most of their time?
  2. What volumes (cases, documents, tickets per month) and what error or rework rate?
  3. Which decisions in the process affect customers, employees, or applicants directly?
  4. What would a 20% improvement be worth, and how would you notice it?

Data and IT owners

  1. Where does the data for this process live, who owns it, and how is it accessed (API, export, DB)?
  2. How current, complete, and consistent is it? Known quality issues?
  3. How are access rights modelled, and can they be enforced per document or row?
  4. What is the deployment standard: Kubernetes, IaC, CI/CD, cloud provider, on-prem GPUs?

Security, legal, DPO

  1. What is the data classification for the sources in scope, and which classes may go to external APIs?
  2. Which AI tools are approved, and how is shadow AI handled?
  3. Is there an AI policy, an AI inventory, or an ISO/IEC 42001 or 27001 management system?
  4. Have DPIAs been done for AI systems? Which supervisory or sector rules apply (central bank, health, public sector)?

End users

  1. Show us the task on screen. What is tedious, what needs judgment?
  2. Where would you not trust an AI suggestion, and why?

2. Data readiness checklist​

Score each source in scope 0–2 per row (0 = blocker, 1 = workable with effort, 2 = ready).

DimensionCheck
OwnershipNamed data owner and steward
AccessProgrammatic access available (API, DB, export) under a service identity
VolumeEnough examples for the use case (documents for RAG, labelled cases for evaluation)
QualityCompleteness, consistency, duplicates, OCR quality for scans, language mix profiled on a sample
FreshnessUpdate frequency known; change feed or incremental export possible
Structure and metadataTitles, dates, versions, authors, document types available for filtering
PermissionsSource ACLs can be mapped to users at retrieval time
SensitivityData classification known; personal data and special categories (GDPR Art. 9) identified
Legal basisPurpose compatible with original collection; legal basis documented
Retention and erasureRetention rules known; deletion can propagate to indexes and derived datasets
ResidencyStorage and processing location known and acceptable (EU, Germany, on-prem)
Ground truthHistorical outcomes or expert answers available to build an evaluation set

Quick profiling on a sample export:

profile_sample.py
import pandas as pd

df = pd.read_parquet("sample/tickets.parquet")
report = pd.DataFrame({
"null_rate": df.isna().mean().round(3),
"distinct": df.nunique(),
"example": df.iloc[0],
})
print(report)
print("duplicate rows:", df.duplicated().mean().round(3))
print("text length p50/p95:", df["body"].str.len().quantile([0.5, 0.95]).tolist())

For document corpora, sample 50–100 files per source and record: file types, share of scanned PDFs, average pages, language, presence of tables, and how many have usable metadata.

3. Use-case scoring matrix​

Score each candidate 1–5 per criterion; multiply by the weight. Agree on weights with the sponsor before scoring.

CriterionWeight15
Business value25%Nice to haveMeasurable cost or revenue impact
Data readiness20%Data missing or inaccessibleData available, clean, permissioned
Technical feasibility15%Unproven techniqueWell-established pattern (RAG, extraction, classification)
Risk and regulatory exposure (inverted)15%Annex III high-risk or special-category dataInternal, low-impact, no personal data
Measurability10%No baseline or ground truthClear baseline and evaluation set
Time to value10%Over 6 monthsPoC in 4–6 weeks
Adoption readiness5%No owner, user resistanceEngaged owner and users
score.py
weights = {"value": .25, "data": .20, "feasibility": .15, "risk": .15,
"measurable": .10, "time": .10, "adoption": .05}

candidates = {
"Contract clause extraction": {"value": 4, "data": 4, "feasibility": 5, "risk": 4, "measurable": 5, "time": 4, "adoption": 4},
"CV screening assistant": {"value": 4, "data": 3, "feasibility": 4, "risk": 1, "measurable": 3, "time": 2, "adoption": 3},
"Internal IT helpdesk RAG": {"value": 3, "data": 4, "feasibility": 5, "risk": 5, "measurable": 4, "time": 5, "adoption": 4},
}

ranked = sorted(((sum(weights[k] * v for k, v in s.items()), name) for name, s in candidates.items()), reverse=True)
for score, name in ranked:
print(f"{score:.2f} {name}")

Plot the shortlist on a value vs feasibility quadrant for the readout. The recommended first use case is usually high-value, high-readiness, and low regulatory exposure; high-risk candidates go to the roadmap with the governance work they need.

4. Governance mapping​

Use three lenses; they complement each other.

FrameworkNatureHow we use it
ISO/IEC 42001:2023Certifiable AI management system (AIMS) standard; Annex A controlsGap analysis against clauses 4–10 and Annex A; plan towards certification if the client wants it
ISO/IEC 42005:2025Guidance for AI system impact assessmentsTemplate for per-system impact assessments (supports 42001 clauses 6.1.4 and 8.4)
ISO/IEC 42006:2025Requirements for bodies certifying 42001Relevant when the client selects a certification body
NIST AI RMF 1.0 + NIST AI 600-1 (Generative AI Profile)Voluntary framework: Govern, Map, Measure, Manage; 600-1 lists 12 GenAI risksRisk register structure and control ideas; check nist.gov for the current status of any AI RMF 1.0 revision
EU AI Act (Regulation (EU) 2024/1689, amended by 2026/1744)Binding lawRole and risk classification per system; obligations and dates

Governance gap checklist (compressed from 42001 and AI RMF):

AreaEvidence we look for
AI policy and rolesApproved AI policy; accountable owner per AI system; RACI
AI inventoryList of AI systems and tools with purpose, data, provider, role, risk class
Risk and impact assessmentDocumented method; assessments per system
Data governanceData quality, provenance, and bias controls for training and input data
Lifecycle controlsRequirements, evaluation, release approval, monitoring, change control
Third-party managementVendor due diligence for AI providers, contract clauses on data use and residency
TransparencyUser disclosures, documentation for affected persons
Human oversightDefined override and escalation paths
Incident managementAI incidents in the incident process, with reporting duties
CompetenceAI literacy training records

5. EU AI Act classification (as of 2026-10)​

Risk classExamplesKey duties
ProhibitedSocial scoring, manipulative techniques, untargeted facial scraping, emotion recognition at work or school (with exceptions)Must not be placed on the market or used
High-risk (Annex III)Recruitment and HR decisions, education access and grading, creditworthiness, essential public services, law enforcement, migrationProviders: risk management, data governance, documentation, logging, conformity assessment. Deployers (Art. 26): use per instructions, competent human oversight, relevant input data, monitoring, logs kept ≥ 6 months, inform workers and affected persons; FRIA (Art. 27) for certain deployers
Transparency (Art. 50)Chatbots, generative content, emotion recognition, deepfakesDisclose AI interaction; mark synthetic content; disclose deepfakes and AI-generated public-interest text
MinimalSpam filters, internal search, most productivity toolsAI literacy measures (Art. 4); voluntary codes
GPAI modelsFoundation-model providersDocumentation, copyright policy, training-data summary; systemic-risk models have more

Timeline after the Digital Omnibus on AI (Regulation (EU) 2026/1744, in force 2026-07-27):

DateApplies
2025-02-02Prohibitions, AI literacy
2025-08-02GPAI model obligations, governance, penalties
2026-08-02Article 50 transparency
2026-12-02Art. 50(2) marking for generative systems placed on the market before 2026-08-02; new Omnibus prohibitions (non-consensual intimate content, CSAM)
2027-12-02Annex III high-risk systems
2028-08-02Annex I high-risk systems (product safety components)
2030-08-02High-risk systems intended for use by public authorities, placed on the market before the application date

Omnibus changes that matter in assessments (per secondary sources; confirm article numbers and wording in the consolidated text): a new Art. 4a permits processing special-category data for bias detection and correction under safeguards. The secondary sources also describe changes to AI literacy (Art. 4) and FRIA (Art. 27) wording; check both against the consolidated text.

In Hungary, Act LXXV of 2025 and Government Decree 344/2025 implement the national AI Act setup (notifying, market surveillance and single-point-of-contact roles; check the decree for the current authority names). The data protection authority NAIH continues to enforce GDPR. Banking clients should also review MNB Recommendation 13/2025 (XII.3.) on the digital transformation of credit institutions for its AI requirements.

6. GDPR and DPIA touchpoints​

TouchpointWhat to check
RoleController / processor split between client, Lieber, and model or cloud providers; DPAs in place
Legal basisPer processing purpose; legitimate interest needs the three-step test (EDPB Opinion 28/2024 confirms it can apply to AI development and deployment)
Purpose limitationReuse of operational data for AI is compatible with the original purpose, or a new basis exists
MinimisationOnly needed fields reach prompts, indexes, logs, and training sets; redaction in pipelines
DPIA (Art. 35)Required where processing is likely high-risk: new technology plus personal data at scale, evaluation or scoring of people, special categories, monitoring employees. AI Act Art. 26(9) points deployers to provider information for this
Anonymity of modelsA model trained on personal data is not automatically anonymous; the EDPB requires case-by-case evidence that extraction and query risks are insignificant
Upstream lawfulnessDeployers must check that a model they use was not developed through unlawful processing, scaled to risk
TransfersProvider locations and sub-processors; prefer EU or sovereign hosting (see Private & Sovereign AI)
Data subject rightsAccess and erasure must reach vector indexes, caches, traces, and fine-tuning datasets
Automated decisions (Art. 22)Human involvement for decisions with legal or similarly significant effects

DPIA screening output: for each shortlisted use case, "DPIA required / recommended / not required" with the reason, so the DPO can plan before the PoC starts.

7. Architecture and security review​

For each existing or planned AI system, review against a fixed set of questions and record findings by severity.

AreaQuestions
Model accessGateway in front of models? Keys centralised, rotated, scoped?
Data flowWhich data reaches which model, where is it hosted, what is logged?
Identity and permissionsRetrieval filters on the user's permissions? Tools run with user-scoped credentials?
EvaluationGolden set, CI eval gate, online monitoring?
OperationsSLOs, tracing, runbooks, rollback path? See AI Productification
Supply chainPinned dependencies and images, model provenance, licence review, SBOM?

Security mapping:

OWASP LLM Top 10 2025Typical assessment check
LLM01 Prompt InjectionUntrusted content (emails, web pages, documents) reaches prompts with tool access?
LLM02 Sensitive Information DisclosureSecrets or personal data in prompts, logs, or retrievable chunks without ACL?
LLM03 Supply ChainUnpinned packages, unverified model downloads, unvetted MCP servers or plugins?
LLM04 Data and Model PoisoningWho can write to the knowledge base or training data?
LLM05 Improper Output HandlingModel output rendered as HTML, executed, or used in queries without validation?
LLM06 Excessive AgencyAgents with write permissions beyond the task; missing approvals?
LLM07 System Prompt LeakageCredentials or access rules embedded in system prompts?
LLM08 Vector and Embedding WeaknessesShared index across tenants or permission levels?
LLM09 MisinformationNo grounding, citations, or user guidance for high-stakes answers?
LLM10 Unbounded ConsumptionNo rate limits, budgets, or max_tokens caps?

For agentic systems, run a threat-modelling session using MITRE ATLAS: walk the tactics from reconnaissance to impact, and pick the agent-specific techniques (for example agent context poisoning, tool credential harvesting, exfiltration via tool invocation). ATLAS moved to monthly date-based releases (YYYY.MM.N, e.g. v2026.09) in 2026 and adds agent techniques frequently, so cite the version used in the report. Agent architecture guidance is in Agentic AI Platforms.

Optional quick test for an existing chatbot, run only with written client approval and against a non-production endpoint:

redteam/promptfooconfig.yaml
targets:
- id: https
config:
url: https://staging.client.example/api/chat
method: POST
body: { "message": "{{prompt}}" }
redteam:
purpose: "Internal HR policy assistant for employees"
plugins:
- owasp:llm
strategies:
- jailbreak-templates
- jailbreak

The HTTP target block shape was not re-verified against promptfoo's provider docs; check it before running.

8. Assemble the report​

report-template.md
# AI Assessment — <Client> — <date>

## 1. Executive summary
- Recommended first use case, expected value, PoC scope, decision needed

## 2. Scope and method
- Units, systems, interviews held, documents reviewed, assessment dates

## 3. Use-case portfolio
- Long list → scoring matrix (weights, scores) → shortlist quadrant

## 4. Recommended first use case
- Problem, users, success metric and baseline, data needed (sources, fields, volume)
- Risk class (EU AI Act), DPIA screening result, hosting option

## 5. Data readiness
- Per-source scorecard, blockers, remediation steps scoped to the use case

## 6. Governance and compliance
- ISO/IEC 42001 and NIST AI RMF gap table, EU AI Act inventory and classification, GDPR findings

## 7. Architecture and security
- Current state, target architecture diagram, OWASP / ATLAS findings by severity

## 8. Roadmap (90 days)
- Workstreams, owners, milestones: PoC, data fixes, governance actions

## Appendices
- Interview notes (anonymised), data profiles, AI inventory, sources

Severity scale for findings: High (blocks the first use case or creates regulatory exposure), Medium (must be fixed before production), Low (improvement).

Best practices​

  • Fix the scoring weights with the sponsor before the workshop; it keeps the ranking credible.
  • Assess data against the chosen use case, not the whole estate; remediation is scoped and iterative.
  • Profile real samples; interview answers about data quality are estimates.
  • Bring the DPO and security lead into week 1, not the readout.
  • Record framework versions and dates (AI Act consolidated text, ATLAS version, OWASP edition) in the report.

Pitfalls to design around​

  • Long lists without decisions: end with one recommended use case and a clear PoC scope.
  • Treating compliance as a final gate: classify risk during scoring so high-risk candidates get realistic timelines.
  • Shadow AI: include SaaS copilots and browser tools in the AI inventory; they carry most of the data-leakage exposure.
  • Outdated deadlines: the Omnibus moved high-risk dates; quote the current timeline and its source.

Definition of done​

  • Interviews completed across sponsor, business, data/IT, security/legal/DPO, and users
  • Data source inventory with readiness scores and profiled samples for shortlisted use cases
  • Scoring matrix with agreed weights; shortlist and one recommended first use case
  • Success metric and baseline defined for the first use case; evaluation data source identified
  • AI inventory with EU AI Act role and risk class per system, using the post-Omnibus timeline
  • ISO/IEC 42001 / NIST AI RMF gap analysis with owners
  • GDPR touchpoints reviewed; DPIA screening per shortlisted use case
  • Architecture and security review with OWASP LLM Top 10 2025 and MITRE ATLAS mapping
  • 90-day roadmap with owners and PoC scope
  • Report and readout delivered; framework versions and sources recorded

Sources​