AI Assessment & Audits
Service: AI Assessment & Audits · Typical shape: 1–3 weeks, fixed scope; 1 lead engineer + 1 data/ML engineer, with a security or compliance specialist for audit-heavy scopes
What we deliver
A decision-ready picture of where the client stands and one concrete first use case with the data it needs, mapped to the regulatory obligations that apply.
| Deliverable | Content |
|---|---|
| Use-case portfolio | Long list, scored shortlist, recommended first use case with success metric |
| Data readiness report | Per data source: owner, access, quality, sensitivity, legal basis, gaps, and remediation steps tied to the chosen use case |
| Governance gap analysis | Current practice vs ISO/IEC 42001 controls, NIST AI RMF functions, EU AI Act role and risk class, GDPR/DPIA needs |
| Architecture and security review | Target architecture sketch; OWASP LLM Top 10 2025 and MITRE ATLAS findings for existing AI systems |
| Roadmap | 90-day plan: PoC scope, data fixes, governance actions, owners |
| Executive readout | 10–15 slides plus the full report |
The output flows directly into a GenAI PoC, and for cloud-side findings into a Cloud & Migration Assessment.
Assessment method
| Week | 1-week variant | 3-week variant |
|---|---|---|
| 1 | Kickoff, 5–6 interviews, data source inventory, long list, scoring workshop | Kickoff, 8–12 interviews, data inventory, document review |
| 2 | — | Data profiling on samples, scoring workshop, governance mapping, security review of existing AI systems |
| 3 | Readout at end of week 1 (light report) | Target architecture, DPIA screening, roadmap, readout |
Day-0 document request
- Org chart for IT, data, security, legal/DPO, and the business units in scope
- Inventory of existing AI/ML systems and AI tools in use (including SaaS copilots)
- Data catalogue or list of core systems (ERP, CRM, DMS, ticketing, data warehouse)
- Information security policies, data classification scheme, records of processing (GDPR Art. 30)
- Existing DPIAs, vendor contracts and DPAs for AI services
- Cloud and infrastructure overview, including GPU availability
Implementation steps
1. Interviews
Run 45-minute interviews with two people from Lieber (one leads, one takes notes into the template). Group by role.
Executive sponsor
- Which business outcome would justify AI investment this year, and how is it measured today?
- What has been tried with AI so far, and what was learned?
- What is the risk appetite for data leaving the company or the EU?
- Who decides on go/no-go for a PoC and for production?
Business process owner
- Walk us through the process step by step. Where do people spend most of their time?
- What volumes (cases, documents, tickets per month) and what error or rework rate?
- Which decisions in the process affect customers, employees, or applicants directly?
- What would a 20% improvement be worth, and how would you notice it?
Data and IT owners
- Where does the data for this process live, who owns it, and how is it accessed (API, export, DB)?
- How current, complete, and consistent is it? Known quality issues?
- How are access rights modelled, and can they be enforced per document or row?
- What is the deployment standard: Kubernetes, IaC, CI/CD, cloud provider, on-prem GPUs?
Security, legal, DPO
- What is the data classification for the sources in scope, and which classes may go to external APIs?
- Which AI tools are approved, and how is shadow AI handled?
- Is there an AI policy, an AI inventory, or an ISO/IEC 42001 or 27001 management system?
- Have DPIAs been done for AI systems? Which supervisory or sector rules apply (central bank, health, public sector)?
End users
- Show us the task on screen. What is tedious, what needs judgment?
- Where would you not trust an AI suggestion, and why?
2. Data readiness checklist
Score each source in scope 0–2 per row (0 = blocker, 1 = workable with effort, 2 = ready).
| Dimension | Check |
|---|---|
| Ownership | Named data owner and steward |
| Access | Programmatic access available (API, DB, export) under a service identity |
| Volume | Enough examples for the use case (documents for RAG, labelled cases for evaluation) |
| Quality | Completeness, consistency, duplicates, OCR quality for scans, language mix profiled on a sample |
| Freshness | Update frequency known; change feed or incremental export possible |
| Structure and metadata | Titles, dates, versions, authors, document types available for filtering |
| Permissions | Source ACLs can be mapped to users at retrieval time |
| Sensitivity | Data classification known; personal data and special categories (GDPR Art. 9) identified |
| Legal basis | Purpose compatible with original collection; legal basis documented |
| Retention and erasure | Retention rules known; deletion can propagate to indexes and derived datasets |
| Residency | Storage and processing location known and acceptable (EU, Germany, on-prem) |
| Ground truth | Historical outcomes or expert answers available to build an evaluation set |
Quick profiling on a sample export:
import pandas as pd
df = pd.read_parquet("sample/tickets.parquet")
report = pd.DataFrame({
"null_rate": df.isna().mean().round(3),
"distinct": df.nunique(),
"example": df.iloc[0],
})
print(report)
print("duplicate rows:", df.duplicated().mean().round(3))
print("text length p50/p95:", df["body"].str.len().quantile([0.5, 0.95]).tolist())
For document corpora, sample 50–100 files per source and record: file types, share of scanned PDFs, average pages, language, presence of tables, and how many have usable metadata.
3. Use-case scoring matrix
Score each candidate 1–5 per criterion; multiply by the weight. Agree on weights with the sponsor before scoring.
| Criterion | Weight | 1 | 5 |
|---|---|---|---|
| Business value | 25% | Nice to have | Measurable cost or revenue impact |
| Data readiness | 20% | Data missing or inaccessible | Data available, clean, permissioned |
| Technical feasibility | 15% | Unproven technique | Well-established pattern (RAG, extraction, classification) |
| Risk and regulatory exposure (inverted) | 15% | Annex III high-risk or special-category data | Internal, low-impact, no personal data |
| Measurability | 10% | No baseline or ground truth | Clear baseline and evaluation set |
| Time to value | 10% | Over 6 months | PoC in 4–6 weeks |
| Adoption readiness | 5% | No owner, user resistance | Engaged owner and users |
weights = {"value": .25, "data": .20, "feasibility": .15, "risk": .15,
"measurable": .10, "time": .10, "adoption": .05}
candidates = {
"Contract clause extraction": {"value": 4, "data": 4, "feasibility": 5, "risk": 4, "measurable": 5, "time": 4, "adoption": 4},
"CV screening assistant": {"value": 4, "data": 3, "feasibility": 4, "risk": 1, "measurable": 3, "time": 2, "adoption": 3},
"Internal IT helpdesk RAG": {"value": 3, "data": 4, "feasibility": 5, "risk": 5, "measurable": 4, "time": 5, "adoption": 4},
}
ranked = sorted(((sum(weights[k] * v for k, v in s.items()), name) for name, s in candidates.items()), reverse=True)
for score, name in ranked:
print(f"{score:.2f} {name}")
Plot the shortlist on a value vs feasibility quadrant for the readout. The recommended first use case is usually high-value, high-readiness, and low regulatory exposure; high-risk candidates go to the roadmap with the governance work they need.
4. Governance mapping
Use three lenses; they complement each other.
| Framework | Nature | How we use it |
|---|---|---|
| ISO/IEC 42001:2023 | Certifiable AI management system (AIMS) standard; Annex A controls | Gap analysis against clauses 4–10 and Annex A; plan towards certification if the client wants it |
| ISO/IEC 42005:2025 | Guidance for AI system impact assessments | Template for per-system impact assessments (supports 42001 clauses 6.1.4 and 8.4) |
| ISO/IEC 42006:2025 | Requirements for bodies certifying 42001 | Relevant when the client selects a certification body |
| NIST AI RMF 1.0 + NIST AI 600-1 (Generative AI Profile) | Voluntary framework: Govern, Map, Measure, Manage; 600-1 lists 12 GenAI risks | Risk register structure and control ideas; check nist.gov for the current status of any AI RMF 1.0 revision |
| EU AI Act (Regulation (EU) 2024/1689, amended by 2026/1744) | Binding law | Role and risk classification per system; obligations and dates |
Governance gap checklist (compressed from 42001 and AI RMF):
| Area | Evidence we look for |
|---|---|
| AI policy and roles | Approved AI policy; accountable owner per AI system; RACI |
| AI inventory | List of AI systems and tools with purpose, data, provider, role, risk class |
| Risk and impact assessment | Documented method; assessments per system |
| Data governance | Data quality, provenance, and bias controls for training and input data |
| Lifecycle controls | Requirements, evaluation, release approval, monitoring, change control |
| Third-party management | Vendor due diligence for AI providers, contract clauses on data use and residency |
| Transparency | User disclosures, documentation for affected persons |
| Human oversight | Defined override and escalation paths |
| Incident management | AI incidents in the incident process, with reporting duties |
| Competence | AI literacy training records |
5. EU AI Act classification (as of 2026-10)
| Risk class | Examples | Key duties |
|---|---|---|
| Prohibited | Social scoring, manipulative techniques, untargeted facial scraping, emotion recognition at work or school (with exceptions) | Must not be placed on the market or used |
| High-risk (Annex III) | Recruitment and HR decisions, education access and grading, creditworthiness, essential public services, law enforcement, migration | Providers: risk management, data governance, documentation, logging, conformity assessment. Deployers (Art. 26): use per instructions, competent human oversight, relevant input data, monitoring, logs kept ≥ 6 months, inform workers and affected persons; FRIA (Art. 27) for certain deployers |
| Transparency (Art. 50) | Chatbots, generative content, emotion recognition, deepfakes | Disclose AI interaction; mark synthetic content; disclose deepfakes and AI-generated public-interest text |
| Minimal | Spam filters, internal search, most productivity tools | AI literacy measures (Art. 4); voluntary codes |
| GPAI models | Foundation-model providers | Documentation, copyright policy, training-data summary; systemic-risk models have more |
Timeline after the Digital Omnibus on AI (Regulation (EU) 2026/1744, in force 2026-07-27):
| Date | Applies |
|---|---|
| 2025-02-02 | Prohibitions, AI literacy |
| 2025-08-02 | GPAI model obligations, governance, penalties |
| 2026-08-02 | Article 50 transparency |
| 2026-12-02 | Art. 50(2) marking for generative systems placed on the market before 2026-08-02; new Omnibus prohibitions (non-consensual intimate content, CSAM) |
| 2027-12-02 | Annex III high-risk systems |
| 2028-08-02 | Annex I high-risk systems (product safety components) |
| 2030-08-02 | High-risk systems intended for use by public authorities, placed on the market before the application date |
Omnibus changes that matter in assessments (per secondary sources; confirm article numbers and wording in the consolidated text): a new Art. 4a permits processing special-category data for bias detection and correction under safeguards. The secondary sources also describe changes to AI literacy (Art. 4) and FRIA (Art. 27) wording; check both against the consolidated text.
In Hungary, Act LXXV of 2025 and Government Decree 344/2025 implement the national AI Act setup (notifying, market surveillance and single-point-of-contact roles; check the decree for the current authority names). The data protection authority NAIH continues to enforce GDPR. Banking clients should also review MNB Recommendation 13/2025 (XII.3.) on the digital transformation of credit institutions for its AI requirements.
6. GDPR and DPIA touchpoints
| Touchpoint | What to check |
|---|---|
| Role | Controller / processor split between client, Lieber, and model or cloud providers; DPAs in place |
| Legal basis | Per processing purpose; legitimate interest needs the three-step test (EDPB Opinion 28/2024 confirms it can apply to AI development and deployment) |
| Purpose limitation | Reuse of operational data for AI is compatible with the original purpose, or a new basis exists |
| Minimisation | Only needed fields reach prompts, indexes, logs, and training sets; redaction in pipelines |
| DPIA (Art. 35) | Required where processing is likely high-risk: new technology plus personal data at scale, evaluation or scoring of people, special categories, monitoring employees. AI Act Art. 26(9) points deployers to provider information for this |
| Anonymity of models | A model trained on personal data is not automatically anonymous; the EDPB requires case-by-case evidence that extraction and query risks are insignificant |
| Upstream lawfulness | Deployers must check that a model they use was not developed through unlawful processing, scaled to risk |
| Transfers | Provider locations and sub-processors; prefer EU or sovereign hosting (see Private & Sovereign AI) |
| Data subject rights | Access and erasure must reach vector indexes, caches, traces, and fine-tuning datasets |
| Automated decisions (Art. 22) | Human involvement for decisions with legal or similarly significant effects |
DPIA screening output: for each shortlisted use case, "DPIA required / recommended / not required" with the reason, so the DPO can plan before the PoC starts.
7. Architecture and security review
For each existing or planned AI system, review against a fixed set of questions and record findings by severity.
| Area | Questions |
|---|---|
| Model access | Gateway in front of models? Keys centralised, rotated, scoped? |
| Data flow | Which data reaches which model, where is it hosted, what is logged? |
| Identity and permissions | Retrieval filters on the user's permissions? Tools run with user-scoped credentials? |
| Evaluation | Golden set, CI eval gate, online monitoring? |
| Operations | SLOs, tracing, runbooks, rollback path? See AI Productification |
| Supply chain | Pinned dependencies and images, model provenance, licence review, SBOM? |
Security mapping:
| OWASP LLM Top 10 2025 | Typical assessment check |
|---|---|
| LLM01 Prompt Injection | Untrusted content (emails, web pages, documents) reaches prompts with tool access? |
| LLM02 Sensitive Information Disclosure | Secrets or personal data in prompts, logs, or retrievable chunks without ACL? |
| LLM03 Supply Chain | Unpinned packages, unverified model downloads, unvetted MCP servers or plugins? |
| LLM04 Data and Model Poisoning | Who can write to the knowledge base or training data? |
| LLM05 Improper Output Handling | Model output rendered as HTML, executed, or used in queries without validation? |
| LLM06 Excessive Agency | Agents with write permissions beyond the task; missing approvals? |
| LLM07 System Prompt Leakage | Credentials or access rules embedded in system prompts? |
| LLM08 Vector and Embedding Weaknesses | Shared index across tenants or permission levels? |
| LLM09 Misinformation | No grounding, citations, or user guidance for high-stakes answers? |
| LLM10 Unbounded Consumption | No rate limits, budgets, or max_tokens caps? |
For agentic systems, run a threat-modelling session using MITRE ATLAS: walk the tactics from reconnaissance to impact, and pick the agent-specific techniques (for example agent context poisoning, tool credential harvesting, exfiltration via tool invocation). ATLAS moved to monthly date-based releases (YYYY.MM.N, e.g. v2026.09) in 2026 and adds agent techniques frequently, so cite the version used in the report. Agent architecture guidance is in Agentic AI Platforms.
Optional quick test for an existing chatbot, run only with written client approval and against a non-production endpoint:
targets:
- id: https
config:
url: https://staging.client.example/api/chat
method: POST
body: { "message": "{{prompt}}" }
redteam:
purpose: "Internal HR policy assistant for employees"
plugins:
- owasp:llm
strategies:
- jailbreak-templates
- jailbreak
The HTTP target block shape was not re-verified against promptfoo's provider docs; check it before running.
8. Assemble the report
# AI Assessment — <Client> — <date>
## 1. Executive summary
- Recommended first use case, expected value, PoC scope, decision needed
## 2. Scope and method
- Units, systems, interviews held, documents reviewed, assessment dates
## 3. Use-case portfolio
- Long list → scoring matrix (weights, scores) → shortlist quadrant
## 4. Recommended first use case
- Problem, users, success metric and baseline, data needed (sources, fields, volume)
- Risk class (EU AI Act), DPIA screening result, hosting option
## 5. Data readiness
- Per-source scorecard, blockers, remediation steps scoped to the use case
## 6. Governance and compliance
- ISO/IEC 42001 and NIST AI RMF gap table, EU AI Act inventory and classification, GDPR findings
## 7. Architecture and security
- Current state, target architecture diagram, OWASP / ATLAS findings by severity
## 8. Roadmap (90 days)
- Workstreams, owners, milestones: PoC, data fixes, governance actions
## Appendices
- Interview notes (anonymised), data profiles, AI inventory, sources
Severity scale for findings: High (blocks the first use case or creates regulatory exposure), Medium (must be fixed before production), Low (improvement).
Best practices
- Fix the scoring weights with the sponsor before the workshop; it keeps the ranking credible.
- Assess data against the chosen use case, not the whole estate; remediation is scoped and iterative.
- Profile real samples; interview answers about data quality are estimates.
- Bring the DPO and security lead into week 1, not the readout.
- Record framework versions and dates (AI Act consolidated text, ATLAS version, OWASP edition) in the report.
Pitfalls to design around
- Long lists without decisions: end with one recommended use case and a clear PoC scope.
- Treating compliance as a final gate: classify risk during scoring so high-risk candidates get realistic timelines.
- Shadow AI: include SaaS copilots and browser tools in the AI inventory; they carry most of the data-leakage exposure.
- Outdated deadlines: the Omnibus moved high-risk dates; quote the current timeline and its source.
Definition of done
- Interviews completed across sponsor, business, data/IT, security/legal/DPO, and users
- Data source inventory with readiness scores and profiled samples for shortlisted use cases
- Scoring matrix with agreed weights; shortlist and one recommended first use case
- Success metric and baseline defined for the first use case; evaluation data source identified
- AI inventory with EU AI Act role and risk class per system, using the post-Omnibus timeline
- ISO/IEC 42001 / NIST AI RMF gap analysis with owners
- GDPR touchpoints reviewed; DPIA screening per shortlisted use case
- Architecture and security review with OWASP LLM Top 10 2025 and MITRE ATLAS mapping
- 90-day roadmap with owners and PoC scope
- Report and readout delivered; framework versions and sources recorded
Sources
- EU AI Act Digital Omnibus (Regulation (EU) 2026/1744), dates and deployer changes: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=OJ:L_202601744 · https://artificialintelligenceact.eu/ai-act-explorer/digital-omnibus/ · https://www.orrick.com/en/Insights/2026/07/EU-AI-Act-Update-Digital-Omnibus-Finalizes-8-Compliance-Changes · https://cdp.cooley.com/digital-ai-omnibus-delays-key-deadlines-introduces-new-rules/
- EU AI Act Article 26 (deployers of high-risk systems): https://artificialintelligenceact.eu/article/26/ · Article 50 (transparency): https://artificialintelligenceact.eu/article/50/
- Hungarian implementation: https://njt.jog.gov.hu/jogszabaly/2025-344-20-22 · AI market surveillance authority: https://mihivatal.gov.hu/rolunk · MNB Recommendation 13/2025: https://www.mnb.hu/letoltes/13-2025-banki-digitalis-ajanlas.pdf · https://digitalpolicyalert.org/event/34952-act-lxxv-of-2025-on-the-implementation-of-the-european-union-regulation-on-artificial-intelligence-in-hungary-enters-into-force-with-grace-period · https://www.regulatoryai.eu/ai-act-hungary/ · https://practiceguides.chambers.com/practice-guides/artificial-intelligence-2026/hungary/trends-and-developments
- ISO/IEC 42005:2025: https://www.iso.org/standard/42005 · ISO/IEC 42006:2025: https://www.iso.org/standard/42006 · 42000-series status: https://gaicc.org/blog/iso-42000-standards-family/ · https://casrai.org/guides/iso-42006-explained-who-can-certify-iso-42001
- NIST AI RMF (revision under way): https://www.nist.gov/itl/ai-risk-management-framework · NIST AI 600-1 overview: https://casrai.org/guides/nist-ai-rmf-generative-ai-profile
- EDPB Opinion 28/2024 on AI models: https://cms.law/en/nld/legal-updates/edpb-opinion-28-2024-key-takeaways-on-processing-personal-data-in-the-context-of-ai-models · https://www.aoshearman.com/en/insights/ao-shearman-on-data/edpb-publishes-opinion-on-personal-data-processing-relating-to-ai-model-development
- OWASP Top 10 for LLM Applications 2025: https://genai.owasp.org/llm-top-10/
- MITRE ATLAS: https://atlas.mitre.org/ · release changelog (date-based versions): https://github.com/mitre-atlas/atlas-data/blob/main/CHANGELOG.md · agentic technique additions: https://dev.to/brennhill/mitre-atlas-now-has-agentic-attack-techniques-3815
- promptfoo OWASP LLM red-team plugins: https://www.promptfoo.dev/docs/red-team/owasp-llm-top-10/