Delphi4All — User manual

What Delphi4All is

Delphi4All is for running a consensus conference with the Delphi method. A group of experts — the panelists — votes on many possible clinical cases; the system works out where there is agreement and where there is not, puts back to the vote only what the group is divided on, and in the end describes the results as briefly as possible.

The two roles

• The coordinator registers on the site, builds the study, invites the panelists, opens and closes the rounds and reads the final report.
• The panelist does not register: they receive a personal access code by email and vote with it. The code identifies their votes, so it must not be shared.

Registering as a coordinator

Registration needs two confirmations and happens in four steps.

1. You fill in the registration form with your name, email address, institution and the reason for the request.
2. An email arrives with a link: opening it confirms that the address is yours.
3. The administrator receives the request and decides whether to approve it.
4. If the request is approved a second email arrives with the link for choosing a password. From then on you can log in.

Your name, institution, language and password can be changed at any time from your profile: your own name in the top bar.

Before you start: what to prepare

A study is easier to build if you sketch it on paper first, and it is worth discussing it with the group before entering it: once the first round has started its structure can no longer be changed. You need three lists.

• The question: what you want to learn from the panel. For example: “when is it appropriate to perform an ultrasound in an adult with acute abdominal pain?”.
• The procedures: the decisions to be judged, that is the possible answers to the question. In the example: ultrasound, abdominal CT, hospital admission.
• The factors that can change the answer: the patient characteristics that, in the experts' view, make a procedure more or less appropriate. In the example: sex, age, pregnancy, fever. They will become the variables.

A tip: include only the factors that someone in the group really believes can change the judgement. Every extra factor multiplies the panelists' work, as shown further on.

Creating the study

From “My studies”, the “New study” box asks for a few things:

• “Title” and “Description”: how the panelists will see the study, at the top of the voting page. It is a good idea to write the study question in the description.
• “Minimum score” and “Maximum score”: the scale. The RAND/UCLA method uses 1 to 9, and it is best left that way: the three areas, low, middle and high, are designed for that scale.
• “Number of panelists”: how many experts you expect. It is a reference; what counts is who you actually add in the Panelists tab.
• “Maximum number of rounds”: two or three are usually enough.

After “Create” the study page opens, divided into tabs: Chapters, Procedures, Panelists, Access matrix, Rounds, Settings, Report. It is best to fill them in that order, which is also the order of the following sections.

Chapters, variables and options

A study is made of chapters. Each chapter holds the variables that describe the case, and each variable has a closed list of options: for instance sex with male and female, age with under 65 and 65 or over, pregnancy with yes and no. The panelist never writes free text: they choose among the options you defined.

A chapter groups cases described by the same variables. If the study covers both adults and children, and different factors matter for children — weight, for example — create two chapters, “Adults” and “Children”. If all cases are described in the same way, one chapter is enough.

How to do it, in the Chapters tab:

1. In “New chapter” type the title and press “Create”.
2. Inside the chapter, in “Name of the new variable”, type the name — for example “Sex” — and press Enter or “New variable”. The cursor stays in the field, so you can type the next variable straight away. The name is enough; the program works out the internal code by itself.
3. Next to each variable, in “Option name”, type its options one at a time, again with Enter: “Male”, then “Female”.

Two rules for options that work. They must exclude each other: a patient cannot fall into two options of the same variable. And they must cover every case: every patient must fall into one. For age, then, “under 65” and “65 or over” work well; “young” and “old” are ambiguous for someone aged fifty.

How many scenarios: watch the growth

The program generates every possible combination of the options. Their number is the product of the number of options of each variable, and it grows very quickly:

• Sex (2) × Age (2) × Fever (2) = 8 scenarios.
• Adding Pregnancy (2): 16 scenarios.
• Splitting age into three bands instead of two: 24 scenarios.
• With five variables of three options each: 243 scenarios.

And every scenario is voted on for every procedure: with 24 scenarios and 3 procedures, each panelist gives 72 votes per round. Under each chapter you can always see how many scenarios will be produced, before and after exceptions. Keep an eye on it as you add variables and options, and ask yourself whether each variable is really worth the work it demands.

Exceptions and scenarios

A scenario is a combination of options, one per variable: for instance sex = male, age = under 65, pregnancy = no. The system generates all the possible combinations of each chapter by itself. Exceptions rule out the impossible ones — sex = male together with pregnancy = yes — which will not appear in the vote.

To create an exception, at the bottom of the chapter pick a value in two or more drop-down lists — for example “Male” under Sex and “Yes” under Pregnancy — leave the others on “—” and press “New exception”. A list left on “—” means “any value”. The scenario count under the chapter drops straight away: in the example from 16 to 12.

When variables, options and exceptions are in place, press “Generate scenarios”. Careful: once the first round has been created the scenarios can no longer be regenerated, because the votes already cast refer to them. Check the list before you begin.

The procedures

The procedures are the questions asked for every scenario: usually few, rarely more than five. In the voting table they are the columns. For each scenario the panelist gives one vote per procedure.

In the Procedures tab type the name and, if needed, a description, then press “New procedure”. Word them as precise decisions that a panelist can judge appropriate or not: “Abdominal ultrasound within 6 hours” works better than “Imaging”. If two procedures are too alike, panelists will give almost identical votes and the result will say little.

Panelists and access codes

The coordinator enters the panelists' names and addresses — usually nine, but the number is free. The system generates a unique code for each and sends it by email together with the link for voting. Anyone who loses the code can have the coordinator send it again.

In the Panelists tab each one needs “Full name”, “Email address” and “Language”: the invitation and the voting page will reach them in that language. “Send invitation” sends the email with the personal link; “Copy link” is for when you would rather send it yourself. Do not send invitations before opening the round: anyone opening the link would find a page with nothing to vote on.

The permission matrix

Not everyone has to vote on everything. The matrix sets out, for each panelist, which chapter and procedure combinations are theirs. A permission given on a chapter-procedure pair holds for every scenario inside that chapter, because it is the scenarios that are voted on, not the chapters.

In the Access matrix tab the rows are the panelists and the columns are the chapter-procedure pairs; a click on a box turns it on or off, then “Save”. Example: if in a mixed panel the radiologist should judge only ultrasound and CT, not admission, turn off their admission box. “Allow all” and “Deny all” act on the whole table.

Boxes you leave untouched follow the “Default permission for cells not set”: that is why a chapter or procedure added later is not left out by oversight. If everyone votes on everything, there is nothing to do.

The settings

The Settings tab holds the consensus rule, its thresholds and the words used in the reports. The default values match the RAND/UCLA method described further on: unless you have a specific reason to change them, leave them as they are.

• “Lowest votes to drop” and “Highest votes to drop”: how many extreme votes to remove before counting. Default: one on each side.
• “Votes outside the band allowed for agreement”: how many votes may fall outside the area without breaking agreement. Default: one.
• “Votes outside the band above which disagreement is strong”: up to this number disagreement is weak, above it is strong. Default: two.
• “Minimum votes to judge a cell”: below this number the cell is not judged. Default: three.
• “Terminology”: the words used for areas and outcomes, for example “appropriate” instead of “high”. They can be changed at any time, even while the study is running.

Final check before the first round

Before involving the panelists, a quick check saves you from having to start over:

1. Every chapter has its variables, and every variable at least two options.
2. The number of scenarios in each chapter is what you expect.
3. You have pressed “Generate scenarios”.
4. All the procedures are there, clearly worded.
5. The panelists have the right address and language, and the matrix says who votes on what.
6. In the Rounds tab: “New round”, then “Open the round”.
7. Only now, in the Panelists tab, send the invitations.

From here the manual goes on with what happens during the voting.

How the voting works

The panelist opens the link they received and finds a table: the first column holds the chapters, split into one row per scenario; the other columns are the procedures. In every cell the numbers of the scale — usually 1 to 9 — are written side by side: you click the number you choose and a circle appears around it. Votes are saved by themselves as you go; at the end you confirm the submission.

When there is agreement

The scale is divided into three areas: low votes, middle area, high votes. For each cell the highest and the lowest vote are set aside — the isolated extremes — and the rest are examined. If they all fall in the same area except at most one, there is agreement. If two fall outside, the disagreement is weak; if more than two, it is strong.

The thresholds, the boundaries of the areas and the words used to name them can be changed in the study settings: they are your choices, not the program's.

The second round

Once the first round is closed, the coordinator can open a second one: only the cells where there was disagreement go back to the vote. Those voting see them again with a horizontal bar under the numbers, running from the lowest to the highest vote given last time, and just below it the median of the previous round. That way everyone sees where the group stood before deciding again.

Agreement formed on the middle area stays agreement and does not go back to the vote: it is a decision of the group, not indecision.

Closing a cell by hand

If a cell will not resolve by itself, the coordinator can close it from the statistics table, with a note and, if wanted, with an outcome decided at the conference. A closed cell does not go back to the vote, and the chosen outcome appears in the final report. Reopening it puts everything back as it was.

The final report

The report does not list the scenarios one by one: it reduces them to the variables that make a difference. If for a procedure agreement holds for males both young and old, the report says «sex = male» and says nothing about age, because it does not change the outcome. For each chapter and procedure the determining and the irrelevant variables are named.

At the top of the report the same areas are written out as sentences, ready to paste into the results section of a paper. Everything can be exported as CSV and as plain text, and the page prints without the navigation.

Languages and the words of the study

The interface exists in several languages and is switched from the top bar: each panelist votes in their own. The words naming the areas and the outcomes — appropriate, uncertain, not appropriate, agreement, disagreement — are not translated: they are data of your study, written in the settings, so they appear the same to everyone and in the final report.