Merge Patient Data Project - GSoC FINAL REPORT

 Project : Merge Patient Data From Multiple Installations

• Primary mentor : Stephen Senkomago Musoke 
• Secondary mentor : Daniel Kayiwa
• Student : Samuel Male
• Project wiki : https://wiki.openmrs.org/display/projects/Merge+Patient+data+from+Multiple+Installations

Overview

There are individual installations where each installation is based at a facility (no guarantee of a consistent Internet connection), and is at the same version (OpenMRS, HTML forms, concepts and other metadata) - but there is a need to bring the patient records (or extracts) together to a central database. Site level users and metadata are not synced, neither are concepts and forms, as they are expected to be similar. The merged data (father instance) would be read-only, used reporting and analysis purposes.

Objectives

• Merge Patients      --> Achieved
• Merge Encounters --> Achieved
• Merge Obs             --> Achieved

Extra Credit

• Encrypt Data During Transfer --> Achieved 

Contributions

While working on this project, I developed a new Openmrs module.
If you wanna use this module, what you simply want is to build this project,  install your .OMOD on your running Openmrs instance and your set.

mergepatientdata --> https://github.com/samuelmale/openmrs-module-mergepatientdata 

While working, I felt comfortable managing my progress using trello, I setup my dashboard before my second evaluation. I worked on the following cards.

• MPD-1
• MPD-2
• MPD-3
• MPD-4
• VM-11

Commits on the github project can be found here.

Resources 

This is a quick documentation on how to deploy the module with a steps and guidelines in a convention of 'How To' approach available here.

Talk Discussions

• https://talk.openmrs.org/t/merge-patient-data-from-multiple-installations-project/18024
• https://talk.openmrs.org/t/overriding-default-multipart-file-size-in-openmrs-module-dev/19333
• https://talk.openmrs.org/t/failed-export-concepts-metasharing-module/19331/2
• https://talk.openmrs.org/t/updating-the-validation-module/19346

Mid-term Video Presentation

A more detailed Presentation

Experience

Well, this was my first time participating in GSoC, and it has been fun working with Openmrs. I have been able to learn lots of stuff, both programing stuff and communication skills. 

With communication I learnt working with a community which is typically OpenSource, which is made up by an ecosystem of volunteers. 

As for programming, I learned of new Libraries like js-grid. On working with js-grid, I ended up learning of Hibernate Pagination. I don't think I can exhaust all new stuff I hit upon while in the Epic. Just know, I learnt lot in, JS, Java, Spring . etc..

I thank Openmrs for participating in such a resourceful a program. This has given lots of confidence and insight to students aspiring to become advanced devs. Me as a First year student pursuing a Bachelor's of Info Systems, I have been greatly motivated that however much I was/am a newbie to everything, I can do some work. I thank my mentors,  Musoke Steve and Daniel Kayiwa. These guys were like parents to me. Due to my lil understanding of most aspects, these guys have always shown me the right path towards how to implement stuff and for long I looked at them like impossible ideals.    

Week 12

Hi all,

This is the last week of the Epic. If you missed last week's update, then you wanna look at it here.

Work done

• The week started off swiftly with a quick fix of last week's Hibernate issue. Wycliff's advice worked like a charm.
• After fixing this, I thought I was set to go after-all the module worked fine with my small Patient dataset. I then resorted to cleaning the codebase, implementing  most of the TODOs.
• As the week was almost ending, I realized something was not fine. I asked the community to help me out test the module on a more realistic hospital dataset, this is when I realized we had Memory issues.
• I had to increase my JVM heap space.
• I had to optimize my codebase.
• Looked for memory leaks. While doing this, I realized I was Merging Concepts which Daniel claimed shouldn't me Merged since they are expected to be similar. However, I'm still blocked from here.   

Challenges Encountered

I didn't expect to have such a blocker by this time, probably the mistake I made was to test out the module on a serious dataset late. Regardless of whether I'm getting nervous and hairy, we should come up with a fix for this.

Before, I used to test out the module on a server with around 60 Patients, and all was cool. Now with a server containing around 2000+ Patients, I hit upon Memory issues. This could be no worry, probably it could have a configurable workaround but its not the case. On exporting data, a file of around 1GB is produced. This far way large!

Looking at the codebase, I tried to hunt down memory leaks and found that Concepts are consuming more memory space than expected. This is fully addressed here.

Question

How can one identify General Concepts(eg- those already include in the Concept Dictionary) from Customized Concepts?

Regards,

Samuel Male

Week 11

Hi all,

This the end of week 11, and we a heading week 12 which basically the last week of work.
If you missed last week's blog, you probably wanna see it here.

Work done this week.

By the end of this week, the module could ably merge Patient, Location, Encounter and Obs resource.

However, I had to tested it on larger dataset of Patient data. I discovered that with large data I faced java.lang.OutOfMemoryError issues. So I had to make sure my code optimizes memory which I think was solved.

I also done more work on the Audition and done a few changes on the UI.

Challenges Encountered

When you download data from a server with many patient, the file is amazingly big. I of recent downloaded a file of 100mb. Now uploading such a file to another server, I ended in such an error

org.springframework.web.multipart.MaxUploadSizeExceededException: Maximum upload size of
75000000 bytes exceeded; nested exception is org.apache.commons.fileupload.FileUploadBase$SizeLimitExceededException

This calls for overriding the 'maxUploadSize ' property of 'CommonsMultipartResolver'

But I have failed to do this in Openmrs module dev and I posted this on talk here.

I'm also facing another Hibernate issue.

Context of the Hibernate Issue

Like for instance you are merging Patients from serverA to B which a new server with Patients. However, data comes with primary keys and foreign keys set yet we use key Generator class of hibernate. Ok I tried as possible to set all these IDs like patientId to null to tell hibernate that this is a new Object. However, there some culprit id values that I could be missing to set to null which make Hibernate think that we are updating it ending up with StaleStateException

Proposed solution

Check the Hibernate logs to see which row Hibernate is trying to updated or retrieve but I don’t know how to turn the Hibernate logging so I started this talk thread but got no help as yet.

How to Contribute?

If your reading this blog, you could want help me on the following critical blockers that are blocking my progress

• https://talk.openmrs.org/t/enabling-hibernate-logging/19140/9
• https://talk.openmrs.org/t/overriding-the-default-multipart-file-size/19193/6

Regards,

Samuel Male

Week 10

Hi all,

Its now the end of week 10 since the program started.  Incase you missed out last week's feed, you could want to check it out here .

Work done this week.

For this week, I have been working on a few Tickets. These are all about Mapping Patient metadata in a MergeAble format. I finished work on  Mapping Concepts,  OBS and Encounters  now am assembling everything to get sense out my work. In a nutshell, am now using the Mapping to add support for Encounters.

Challenges Encountered.

In every development stage, there are always challenges and its those challenges that make one learn.

Handling database Table Ids for Merged Data 😕 

For instance we have two servers ie:- Server A and B. A has 50 patients while B has 5 Patients. Regardless of the DBMS, Every row needs an Id. Well while merging data, we could want to merge a Patient from the last row of server A to B. Let me assume that the Patient has an Id of 50. When this Patient is merged, the Patient in Server B may look something like below

Id	patient_name
5	Male Samuel
6	Kenny
7	Kaddu Ronnie
8	Another One
9	Some Patient
50	From Server A

Like you can see, it looks dirty within the database.

Proposed Solution

We could first make first make a check to the database to see whether the Patient we are trying to Merge already exists or not, if it exists, the update it otherwise set the id to null and register it as a new Patient. For a few metadata, this is simple but with other metadata like Encounters, Obs, Locations and Concepts that reference the original Patient id it becomes hard to handle this. You could want to advise me on this modal. Please leave a comment of advice.

Regards,

Samuel Male

Week 9

Hi all,

This is the 9th week since the game started. If you missed out about last week's update, you could want to check it out here .

Work Done this week

This was the week were I had my second evaluation. I was making ends meet to seeing myself pass the evaluation. And do you what, I passed the evaluation and thanks to my mentors 😊 

Currently, the module codebase ain't found in the Openmrs repo. Its still within my repo but Steven told me we will just transit it later to the Openmrs repo. So I couldn't use JIRA for PM. But just as a good practice, I came up with a Trello dashboard(MPD) to help me keep track of progress.

So currently I have been working on Mapping Concepts and 85% of the work is complete.

After that I will proceed to Mapping OBS. Note that these are all subtasks for Adding Support for Encounters as a Resource.

Challenges Encountered

I have faced a challenge of Mapping Concepts since they are soo wide and have a complex hierarchy.

I need your tech support in this. But what I believe is that my current implementation(pending) will be able to give the expected performance. What confuses me is "I wonder how to distinguish a Concept from the dictionary from a customized one! "

Otherwise, I expect significant progress due this week.

Regards,

Samuel Male

Week 8

Hi all,

This is the 8th week since I started work on  Merge Patient Data project. I'm very excited to inform you dear reader that we currently have a working module in place, unless otherwise.

Work done this week

Generally, I capitalized from the front end.

• Audition

I managed to put in place a page that lists down all audits that are available. Basically an audit is a brief accountability of what really happened in a given MPD Operation(Export/Import).





I also put came up with a details page for a give audit.

week 7

Hi folks,

Its now seven weeks down the road since I begun working on the Merge Patient Data Module.
I think I have been slow with progress. This has been cause of some inevitable blockers. However, lots of work has been done this week.

Work Done

• I set up my dev environment on a new Platform(mac OSX). Though my performance decreased, I spent sleepless nights configuring stuff and getting used to the platform.

• More development. Last week I had a bug that blocked me for a while. I'm serializing the data using the  gson library. However I was facing a challenge dealing with generic types. I realized I invested more time trying to fix the bug than moving forward.

Suggested Solution

Just for the use case of serialization of data, I just implemented a dataStore

public class MPDStore {
 public MPDStore() {}
 List<Patient> patients;
 List<Location> locations;
 List types = new ArrayList<>(); 
 public List getTypes() { 
 return types; 
 } 
 public void setTypes(List types) { 
 this.types = types; 
 } 
 public List getPatients() { 
 return patients; 
 } 
 public void setPatients(List patients) { 
 this.patients = patients; 
 } 
 public List getLocations() { 
 return locations; 
 } 
 public void setLocations(List locations) { 
 this.locations = locations; 
 } 
 public void addType(MergeAbleDataCategory type) { 
 this.types.add(type); 
 }
 }

With this model, the module perfectly exports and imports Patient data from one Openmrs instance to another.
Currently I'm working on the audition of the module Operations. I anticipate that this won't take me long then I proceed to the Configuration Services.

I'm looking forward to finishing with both the Audit and Configuration Service implementation by 9th this month.

By the way I'm excited to inform you that the module currently offers the basic functionality inline with the project objectives though more work is still needed at the UI to improve the end User Experience.