Everything between the upload and the report
One collection in. Seven test categories out — generated automatically, run as every role that uses your API, delivered in four formats. Here is each moving part.
Start from the file you already have
No agents, no SDKs, no rewrites. BugVilla reads the formats your API is already described in.
Parsing that captures everything a test needs
Every endpoint is normalized into one internal model: method, URL, path and query params, headers, body schema, auth type, and declared examples. Every generator downstream works from that model — so what the parser misses, no test can check.
Postman Collection v2.1 OpenAPI 3.x / Swagger 2.0 Raw cURL & payloads
- JSON or YAML. OpenAPI and Swagger specs parse either way.
- No spec yet? Fine. Paste cURL commands or bare request payloads and start from those.
- Examples preserved. Declared examples seed realistic request data instead of placeholder noise.
One credential set per role that uses your API
A single login can answer "does it work?" It can never answer "can the wrong person reach it?" — so BugVilla tests as Admin, Member, and an unauthenticated stranger.
Detection first, confirmation always
After an upload, BugVilla reads the parsed endpoints and works out what to ask you for. Nothing is persisted until you confirm it.
- Login endpoint, found. Scored on its path and whether its body is password-shaped — a path alone can't separate /auth/login from /auth/me.
- Token path, extracted. Where the token appears in the response (data.accessToken), read from the declared example or schema.
- Roles, suggested. Role words in folder names and tags become the suggested persona list.
- One login per run. Each persona authenticates once per run, not per case — 400 endpoints don't open 400 sessions.
- Strict isolation. Credentials are never shared between personas. Run-scoped ids are — so a Member can probe exactly what an Admin created.
login
Calls the confirmed login endpoint at run start and extracts a fresh token from the response. New token every run — never stale.
static
Paste a long-lived token, API key, or cookie. BugVilla stores it encrypted and sends it as-is.
none
Sends nothing, on purpose. The unauthenticated baseline that proves a protected endpoint is guarded at all.
Seven categories, generated the moment parsing finishes
Suites run in a deliberate order — functional → contract → authorization → negative → boundary → security → performance. Functional first so ids exist to chain; performance last, against a warm service.
Functional
A happy-path request per endpoint, asserting a 2xx. Runs first so the ids it creates exist for everything downstream.
Contract
Every response validated against its JSON Schema with AJV. Schema drift fails loudly, not quietly.
Authorization
The access matrix: every persona replayed against every protected endpoint, asserting allow or deny. Detailed below.
Negative
Missing required fields, wrong types, absent and invalid auth. Your 4xx handling, exercised on purpose.
Boundary & fuzz
Min and max values, oversized payloads, fuzz strings — the inputs real users eventually send.
Security
Auth-bypass probes, injection payloads, and header checks against every endpoint.
Performance
Latency baselines asserted per endpoint. Runs last, against a warm service, so the numbers mean something.
The access matrix
The one generator that is collection-wide rather than per-endpoint — a result only means something next to the other personas'.
Every protected endpoint × every persona
BugVilla replays each protected endpoint as each persona and asserts the expected outcome — allow or deny. Results roll up into an endpoint × persona grid, stored on the run and rendered in the app and every downloadable report.
- Exposures surface instantly. A red cell means a persona reached something it shouldn't — the finding you fix first.
- Broken grants too. A role that should get in but can't is a bug as well — just a different one, reported separately.
- Real ids, real probes. Resources created by Admin in the functional pass are exactly what other personas probe against.
Deliberately asymmetric
The access assertion doesn't treat allow and deny the same way — because their failure modes aren't the same.
| Expectation | Fails on | Why |
|---|---|---|
| deny | any 2xx | The persona got in. That is the vulnerability. |
| deny | any 5xx | The request reached handler code before the permission check. |
| allow | 401 / 403 only | A 404 or 422 usually means bad test data, not bad permissions — failing those buries real findings under setup noise. |
Upload v2 without losing what v1 proved
A new upload appends to a lineage, never replaces it. Every version keeps its own endpoints, suites and runs — so a pass-rate drop is attributable to a specific document.
A lineage, not an overwrite
Endpoints are matched across versions by METHOD /path — the only identity that survives a re-export, since operationIds and Postman item ids are regenerated freely.
- Fingerprint, not diff noise. The hash covers only what a test depends on — params, body schema, response schemas, auth. A docs-only re-export shows zero changes.
- Breaking, precisely defined. A removed endpoint, a newly required param, a changed type, a dropped success status, an altered auth scheme — anything that can invalidate a passing test.
- Contract diff, any two versions. Compare v1 against v4 directly, not just neighbours.
- Personas carry forward. They belong to the project, not the collection — a v2 upload needs no re-setup.
One dataset, four formats
PDF and HTML for people, CSV and JSON for machines — all four built from the same report data, so a finding can't appear in one and not another.
Findings sorted by what to do about them
PDF HTML CSV JSON
Every run produces totals, pass rate, duration, the slowest endpoints, and a per-persona breakdown — with the request, the response, and the assertion result on every case.
- Exposures vs wrongly blocked. "Reached something it shouldn't" — the vulnerability — is reported apart from a broken grant. One list would bury the former.
- Run-to-run diff. Fixed, broken, still failing — what actually changed since the last run.
- Trends. Pass rate and latency across historical runs, not just today's snapshot.
- Self-contained HTML. No external CSS, fonts or scripts — the report gets emailed and opened offline, and still renders.
The unglamorous parts, handled
Scheduled runs
Cron-style recurring runs. Put the suite on a schedule and let regressions surface themselves instead of waiting for a release day.
Environments & secrets
Persona credentials and environment auth are AES-GCM encrypted at rest, masked on read, and scrubbed from stored request/response captures. How secrets are handled.
Re-run any version
Re-run any version's exact configuration at any time — same endpoints, same suites, same personas. Reproduce last month's result today.
Your first matrix is one upload away
Start free, point BugVilla at a collection you already have, and see what every role can really reach.