# FHIR servers for testing

This repository contains setup script for FHIR resource servers widely in use. It comes with a few test scripts to validate functionality and behaviour of server software.

More specifically we used this repo to see if operations like [$export](https://build.fhir.org/ig/HL7/bulk-data/export.html) and [FHIR Documents](https://build.fhir.org/documents.html) return similar results to advice MedMij participants in querying their resource server.


## Resource servers

Five server configurations are included in this repository. The servers are started in containers with Docker Compose and are accessible through the following URLs. For testing purposes, authentication and secure communication are taken out from configuration. Medplum however, requires authentication and LinuxForHealth requires HTTPS.


| Name                 | FHIR capabilities, API endpoints           | Web application, documentation                |
| -------------------- | ------------------------------------------ | --------------------------------------------- |
| Net tools            |  net_tools                                 | https://hub.docker.com/r/someguy123/net-tools |
| Microsoft SQL        |  :1433                                     |                                               |
| PostgreSQL           |  :5432                                     |                                               |
| Redis                |  :6379                                     |                                               |
|                      |                                            |                                               |
| HAPI                 | GET http://localhost:8100/fhir/metadata    | http://localhost:8100/                        |
| Medplum *            | GET http://localhost:8200/fhir/R4/metadata | https://github.com/medplum/medplum/tree/main/packages/app |
| Firely Server        | GET http://localhost:8300/metadata         | http://localhost:8300/                        |
| Microsoft FHIR       | GET http://localhost:8400/metadata         |                                               |
| IBM LinuxForHealth * | GET https://localhost:9443/fhir/metadata   | https://localhost:9443/openapi/ui/            |

_* Note that we couldn't take out auth and https for Medplum and LinuxForHealth, but scripts are updated to work with tokens and local signed certificates. It's a bit of a grind in testing now. We'll get around that if more testing requires it._

## Start servers

Servers are configured in containers with [docker-compose.yaml](docker-compose.yaml). Use [docker-up.sh](docker-up.sh) and [docker-down.sh](docker-down.sh) scripts to start and stop the servers.


![Docker Compose](./docs/readme_docker_compose.png)


## Testdata folder with examples

Examples taken from Nictiz repositories with slight modifications like taking off the `<?xml />` headers to simplify import. For quick testing, the `StructureDefitinon`s are not imported, which should be in a future dataset to fully test MedMij compliancy. Even on permissive settings Firely Server applies validation well and stops importing when fields like `Observation.comments` are in the `*-examples.xml` files. Other servers like HAPI and Microsoft are more forgiving when validation is turned off (by default).

For this first round of tests, these examples will do. For other tests, we should review the dataset. Also, examples may benefit from an update to have resources link to a single Patient doing an $export operation, etc.

- Source: [Nictiz-STU3](https://github.com/Nictiz/Nictiz-STU3) and optionally R4 repositories.
- Import: [Test data](./testdata) folder.

| Patient resource                      | File name                            | Name             |
| ------------------------------------- | ------------------------------------ | ---------------- |
| Patient/medmij-bgz-patient-01         | medmij-bgz-patient-01.xml            | Thomas van Beek  |
| Patient/example                       | medmij-bgz-patient-02-example-id.xml | Ties van Oest    |
| Patient/medmij-medication-patient-01  | medmij-medication-patient-01.xml     | Piet Jansen      |


## Import testdata

Medplum requires an authentication token to import data, some additional scripting would be useful to make this easy. Haven't looked into yet as we quickly want some first results. LinuxForHealth currently fails to connect, look into that later.

```bash
  Usage: import.sh <server> [auth_token]

  ./scripts/import.sh hapi
  ./scripts/import.sh medplum dXlBleGFtcGxlLmNvbTpzZNyZXQ=
  ./scripts/import.sh firely
  ./scripts/import.sh microsoft
  ./scripts/import.sh linuxforhealth
```

[Show video import datasets](./docs/readme_import_datasets.mp4)

![Import datasets](./docs/readme_import_datasets.png)


# Export testdata with script

Export runs an `$everything` and `$export` operation on the FHIR server. Tests with FHIR Documents too, and tests if pagination would work on these operations if requested.

```bash
  Usage: export.sh <testname> <server> [auth_token]

  ./scripts/export.sh test1 hapi
  ./scripts/export.sh test1 medplum mytesttoken
  ./scripts/export.sh test1 firely
  ./scripts/export.sh test1 microsoft
  ./scripts/export.sh test1 linuxforhealth
```

Compare [test results](./testresults/) between FHIR servers.

| Test      | Description |
| --------- | ----------- |
| test01    | Export resources with [Patient $everything operation](https://www.hl7.org/fhir/operation-patient-everything.html).  |
| test02    | Is pagination supported on Patient `$everything` operation? |
| test03    | Export resources with `Patient/id/$everything` operation.  |
| test04    | Is pagination supported with Patient/id/$everything`?_count`? |
| test05    | Export resources with [$export operation](https://build.fhir.org/ig/HL7/bulk-data/export.html).  |
| test06    | Is pagination supported with Patient/id/export? |
| test07    | Export resources with [FHIR Documents](https://hl7.org/fhir/documents.html).  |
| test08    | Is pagination supported with FHIR Documents? |
| test09    | Can we modify $everything output with _include? |
| test10    | Can we modify $everything output with _include fields? |

Results:

- Firely: GET http://localhost:8300/Patient/$everything 501 Not implemented


# Compare tool

Manual comparisation of the ouput is a bit tough and the queries doesn't cover all options. For example, an export can use `_type` parameter  to include resources. Authentication support, comparing number of resources, etc. We wrote a small tool in C# to help compare details for FHIR servers.

```bash
  Linux:   compare
  Windows: compare.exe

  Import:  compare.exe --import
           compare.exe --import firely

  Compare: compare.exe --compare
```
Source code is in the [tools/compare](./tools/compare) folder. Compiled version is in the [tools/compare/bin](./tools/compare/bin) folder, but for this quick one-time execise just running it from Visual Studio is fine.

Here's an example import of testdata on a single server. It takes about 4 minutes for the full set on all servers. Usually, we would import on all servers as a one-time job after `docker-up.sh` and keep using the databases of FHIR-servers until `docker-down.sh` and `clean.sh` removes the data folders (e.g. `servers/postgres/data`). You may choose to pause the FHIR servers as a group and hold on to the data for later use. That saves you time experimenting with the servers.

```
compare.exe --compare hapi


Importeren Nictiz-testdata op HAPI:

Importeren van 744 definities voor profielen, codelijsten enz.

 - [ 1/744] Definitie: ext-AdditionalCategory.xml
 - [ 2/744] Definitie: ext-Context-EpisodeOfCare.xml
 - [ 3/744] Definitie: ext-CopyIndicator.xml
 - [ 4/744] Definitie: ext-EdifactReferenceNumber.xml
 - [ 5/744] Definitie: ext-Vaccination.PharmaceuticalProduct.xml
 - [ 6/744] Definitie: nl-core-AbilityToDressOneself.xml
 - [ 7/744] Definitie: nl-core-AbilityToDrink.DrinkingLimitations.xml
 - [ 8/744] Definitie: nl-core-AbilityToDrink.xml
 - [ 9/744] Definitie: nl-core-AbilityToEat.EatingLimitations.xml
 - [10/744] Definitie: nl-core-AbilityToEat.xml
 - [11/744] Definitie: nl-core-AbilityToGroom.xml
 - [12/744] Definitie: nl-core-AbilityToPerformMouthcareActivities.Prosthesis.Product.xml
 - [13/744] Definitie: nl-core-AbilityToPerformMouthcareActivities.Prosthesis.xml
 - [14/744] Definitie: nl-core-AbilityToPerformMouthcareActivities.xml
 - [15/744] Definitie: nl-core-AbilityToUseToilet.MenstrualCare.xml
 ...
 
 Importeren van 252 voorbeelden.

 - [ 1/252] Voorbeeld: nl-core-AbilityToPerformMouthcareActivities.Prosthesis.Product-01.xml
 - [ 2/252] Voorbeeld: nl-core-AbilityToDressOneself-01.xml
 - [ 3/252] Voorbeeld: nl-core-AbilityToDrink-01.xml
 - [ 4/252] Voorbeeld: nl-core-AbilityToDrink.DrinkingLimitations-01.xml
 - [ 5/252] Voorbeeld: nl-core-AbilityToDrink.DrinkingLimitations-02.xml
 - [ 6/252] Voorbeeld: nl-core-AbilityToEat-01.xml
 - [ 7/252] Voorbeeld: nl-core-AbilityToEat.EatingLimitations-01.xml
 - [ 8/252] Voorbeeld: nl-core-AbilityToEat.EatingLimitations-02.xml
 - [ 9/252] Voorbeeld: nl-core-AbilityToGroom-01.xml
 - [10/252] Voorbeeld: nl-core-AbilityToPerformMouthcareActivities-01.xml
 - [11/252] Voorbeeld: nl-core-AbilityToPerformMouthcareActivities.Prosthesis-01.xml
 - [12/252] Voorbeeld: nl-core-AbilityToUseToilet-01.xml
 - [13/252] Voorbeeld: nl-core-AbilityToWashOneself-01.xml
 - [14/252] Voorbeeld: nl-core-AdvanceDirective-01-Condition-01.xml
 - [15/252] Voorbeeld: nl-core-AdvanceDirective-01-RelatedPerson-01.xml
 ...
 ```

 After an import, the tool can compare results from the FHIR servers, show that in a markdown table, printed here:

 ```
compare.exe --compare

Vergelijken van testresultaten:

 - Session met HAPI
 - Session met Firely
 - Session met Microsoft
 - Session met Medplum
 - Session met IBM
```

 - Session met HAPI
 - Session met Firely
 - Session met Microsoft
 - Session met Medplum
 - Session met IBM


Zijn de FHIR-servers beschikbaar in lokaal netwerk?

| Server                         | Beschikbaar                    |
| ------------------------------ | ------------------------------ |
| HAPI                           | online                         |
| Firely                         | online                         |
| Microsoft                      | online                         |
| Medplum                        | online                         |
| IBM                            | running, outdated              |


Welke export queries voor een externe client beschikbaar?

| Capabilities HAPI         | Beschikbaar               |
| ------------------------- | ------------------------- |
| everything                | beschikbaar               |
| patient-everything        | beschikbaar                          |
| export                    |                           |
| patient-export            | beschikbaar               |
| group-export              | beschikbaar               |
| anonymized-export         |                           |

| Capabilities Microsoft    | Beschikbaar               |
| ------------------------- | ------------------------- |
| everything                |                           |
| patient-everything        | beschikbaar               |
| export                    | beschikbaar               |
| patient-export            | beschikbaar               |
| group-export              | beschikbaar               |
| anonymized-export         | beschikbaar               |

| Capabilities Firely       | Beschikbaar               |
| ------------------------- | ------------------------- |
| everything                | beschikbaar               |
| patient-everything        | beschikbaar               |
| export                    | beschikbaar               |
| patient-export            | beschikbaar               |
| group-export              | beschikbaar               |
| anonymized-export         | beschikbaar               |


Wat geven de servers terug als we ze uitvragen?

| Operations HAPI                | Resources                      |
| ------------------------------ | ------------------------------ |
|  1. $everything                | 79                             |
|  2. $everything _count         | 79                             |
|  3. $everything id             | 59                             |
|  4. $everything id _count      | 59                             |
|  5. $export                    | -                              |
|  6. $export id                 | -                              |
|  7. $document                  | -                              |
|  8. $document id               | -                              |
|  9. $everything id _include    | 59                             |
| 10. $everything id _includes   | 59                             |
| 11. $everything id _include=*  | 59                             |
| 12. $everything id _type       | -                              |

| Operations Microsoft           | Resources                      |
| ------------------------------ | ------------------------------ |
|  1. $everything                | -                              |
|  2. $everything _count         | -                              |
|  3. $everything id             | 191                            |
|  4. $everything id _count      | 192                            |
|  5. $export                    | -                              |
|  6. $export id                 | -                              |
|  7. $document                  | -                              |
|  8. $document id               | -                              |
|  9. $everything id _include    | 212                            |
| 10. $everything id _includes   | 212                            |
| 11. $everything id _include=*  | 192                            |
| 12. $everything id _type       | 168                            |

Firely fully supports all FHIR operations defined by HL7 after a license key is obtained. Plugins and configuration of plugins decide what resources the server returns in operations like $everything. Unfortunately, we most probably cannot override that configuration on a per-query basis with _includes. (Same as for Microsoft, see table above. HAPI does allow adjusting the $everything export with query _includes.)



## Good support for $everything operation, but differences in behaviour

Remarks 2024-02-18 on availability of operations in FHIR servers we're looking at here:

- **HAPI JPA** has [full support](https://smilecdr.com/docs/fhir_repository/reading_data.html#everything-operation) for the $everything operation. HAPI also has interesting alternative ways of passing multiple logical IDs out-of-spec, not relevant to our tests. By default bundles all resource types linked to Patient. I found HAPI to not support (the out-of-specs) pagination on $everything (_count), there is support for $export (which didn't work in this local install, but should be possible) and $document operations not available. There is mention of Bulk Data Export (= $export) in changelogs and code.
- **HAPI Plain** might be in use at PGO as a more flexible FHIR-interface built-in on other software (maybe for testing purposes). Good to know is that [HAPI Plain](https://hapifhir.io/hapi-fhir/docs/server_plain/introduction.html) won't support $everything operation and others out-of-the-box: requires a manual operation method, but fairly simple to implement.
- **Firely Server** has [full support](https://docs.fire.ly/projects/Firely-Server/en/latest/features_and_tools/custom_operations/patienteverything.html) of $everything and other operations, and it requires a non-trial-licence and turning on a plugin at `appsettings.default.conf`. This took me by surpise and makes it difficult to test for now: we may request a temp licence with these plugins allowed for further testing. Didn't do that for now. Firely supports $export and $document operations too. Firely by default *filters out Practitioner* from $everything for privacy considerations, but there are configuration options for which resources to include with the Patient compartment. We can manipulate includes in the client query with _include and _revinclude too, but couldn't test if this overrides the server settings. Most likely takes a secure approach and prevents if not configured.
- **Microsoft FHIR Server** has [full support](https://learn.microsoft.com/en-us/azure/healthcare-apis/fhir/patient-everything) for the $everything operation. Unfortunately, the server does not support _include on $everything operation. The newer server version is named 'Azure API for FHIR' and lets the system level Bulk [$export operation](https://learn.microsoft.com/en-us/azure/healthcare-apis/fhir/export-data) export to Azure Blob Storage, which would make $export less suitable for a scenario of transferring data between PGOs.
- **Medplum FHIR** [supports](https://www.medplum.com/docs/api/fhir/operations/patient-everything) the Patient/id/$everything syntax. Supports $export, probably $document operations too. The [Composition-document bundle](https://build.fhir.org/composition-operation-document.html) is built-in the source code, so $document should work too. Haven't tested, but it means that Medplum at least supports returning of document bundles.
- **IBM LinuxForHealth** [supports](https://github.com/LinuxForHealth/FHIR) the $everything operation. While it mentions the Java API is not stable, this just works fine. The project however didn't merge updates for a couple of years, nor communicated on status. EOL.
- Adding _include and _revinclude for Patient [$everything operation compartment](https://www.hl7.org/fhir/compartmentdefinition-patient.html) would have been a nice way to make sure all resources are included in the export. Unfortuntaley, Microsoft's implementation doesn't support this and Firely's implementation most likely requires fixed configuration in appsettings.

**Summary:** 

Trying to find a way for servers to behave similar in exporting data, a solution seem to increasingly require more custom work that just calling an endpoint when you look at behaviour of FHIR-servers.

1. The `Patient/$everything` operation is NOT widely available.
2. The `Patient/id/$everything` operation IS widely available. Unfortunate drawback is that different servers behve differently when including linked resources. Firely supports configuration of what exactly serverwide, HAPI returns all within Patient compartment, etc. Not all servers support _includes and _pagination on this operation. That is very unfortunate, because this way it is hard to get similar behaviour in exports of a patient's data.
3. Other export operations may have a server specific implementation too: Bulk Export operations and Compartment Document bundles are well supported by most servers, but trying to call these with an external PGO client will not always lead to a returned bundle. Microsoft's implementation exports to Azure storage only.
4. Most FHIR-servers allow you to simply built an operation plugin or such. That's an inconvenience, we would be asking for a $everything_medmij operation. That comes down to writing a small API.

Failing a.t.m. to find a single FHIR operation (through query parameters and server configuration) that behaves similar over server brands. We may have to resort to a custom operation or API returning a search bundle or FHIR Documents bundle. Probably that would be an acceptaptable amount of work for a PGO to implement, but it would be nice if we could find a way to do this in a single query with the standard FHIR operations.

**Alternatives to Patient-export in full:**

We're discussing this in ontwerpdocument and in team, but quick overview:

1. Receiving PGO runs a `GET Patient/id/$everything` with source PGO and accept that there are differences in completiness of levensloopdossier. Best effort.
2. Receiving PGO runs [multiple queries for gegevensdienst](https://informatiestandaarden.nictiz.nl/wiki/MedMij:V2020.01/Kwalificatie) or [all resources linked to Patient compartment]( https://www.hl7.org/fhir/compartmentdefinition-patient.html) with source PGO.
3. Receiving PGO calls a single API-endpoint on source PGO that returns a bundle of all resources linked to Patient compartment. Similar as (2) having the receiving PGO run queries. Now the source PGO is responsible for creating a bundle.

There is an advantage of shifting the responsibility for bundling to source (2 -> 3), so source has control over what data is sent over. There's more control than with direct FHIR queries from receiving PGO to. And quite useful: it wouldn't require source PGO to offer a full FHIR server.
 
# Net tools

Some notes helpful in testing connections with containers in the local Docker subnet.

```bash
  su -
  apt update
  apt install -y postgresql-client-common postgresql-client redis-tools

  nc -zv sql_postgres 5432
  telnet sql_postgres 5432
  pg_isready -h sql_postgres -p 5432
  psql -h sql_postgres -p 5432  -U medplum -d medplum
```