Slow display of lab reports

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow display of lab reports

Daniel Ek
Can anyone provide advice on the problem of intermittently having issues with extremely slow  display of lab reports. These are HL-7 files I believe which are received from our hospital and from Lifelabs. They populate the Inbox of OSCAR (version 12.1) and when clicking to view, can take more than a minute to display, and at times the "system" will "time out" and fail to display. This happens whether one is accessing from the Lab file of the patient e-chart, or from our individual Inboxes.

This does not happen with Documents which are reports scanned into OSCAR.

There is some correlation with the time it takes to open and the length of the report, but frequently one is unable to open a simple lab test of one line only (ex. random glucose). I have had some success by clicking on other lab results in quick succession, and eventually one will open. And sometimes by returning to the original which did not open, it will open unexpectedly quickly.

I do not know anything about the server we currently use. Our OSP is Indivica, and they cannot define a specific reason for the problem, as it is not consistent.

Dan Ek
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Oscarmcmaster-ontario-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/oscarmcmaster-ontario-users
Reply | Threaded
Open this post in threaded view
|

Re: Slow display of lab reports

Peter Hutten-Czapski-2
Hi Daniel

Welcome to OSCAR users list from the other half of Northern Ontario!

Before I add my $0.02 about tuning OSCAR I should mention that the bc list is better monitored and is more active... even for Ontario specific issues (including lol the Indivica fork) so I have CC'd there

First of all are you slow?
Oscar naturally has several areas where it can be slow.  The inbox, I believe contributed by the same Indivica so they should know it, is often the area where this is seen, however chart opening can be slow.

My benchmarks for remote access is 12 seconds for a chart open (albeit a chart with 10+ years of OSCAR data filling all the boxes)
6 seconds for Inbox to load the list of all new results and HRM reports
an additional 20 seconds for preview to load for the first time
and an additional 3 seconds for preview to load the next bunch of reports when you reach the end of preview

Tuning OSCAR for speed
===================

More or less in increasing orders of complexity.  Take with a grain of salt as you are getting this from a physician, and not a systems operator type.

1) Physical Memory.  If the server is running out of memory then the operating system will write memory to hard disk which substantially and abruptly slows the server.  Look at Admin > System Reports > System status for the box called vmstat
$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0 697952 974068 516076 21689712    0    0   102   331    3    3 15  2 82  1  0

In the above example there is plenty free (for more details google vmstat) so this system is not currently running into an issue
But if it was physical memory is cheap and easy to add to your machine

2) Java Memory allocation to Tomcat Even if you have lots of physical memory if Java is not allowed to use it it will start using the hard drive with the same symptoms. 
Adding memory will not solve this, you have to increase the allocation. There are many ways to determine how much memory Java is using.

In OSCAR 15 enable monitoring (see http://oscarmanual.org/15/5.0/monitoring ). It has lots of performance data (including how much Java memory is being used)
on the same well resourced box as in 1) the data is currently as below
Java memory used: 746 Mb / 7,897 Mb
Its fine (at this time)  However if it wasn't adjust this in /etc/default/tomcat6 
(for OSCAR 15 its in /etc/default/tomcat7)
As a rule of thumb (if you lack this data) ensure that a good 60% - 70% of physical memory is allocated to Java.

3) MySQL itself. There are books on tuning it but for most you can run a tool called mysql tuner Simply.
apt-get install mysqltuner
The script will analyse your MySQL instance and suggest settings for 
/etc/mysql/my.cnf
4) Oscar indices.  As in a library databases use indexes to quicky find the information you want.
You can determine if the queries are running quickly by using the  monitoring tool. Any query running more than half a second (500ms) is noticable to the end user.  
Some time has been spent in OSCAR 15 to optimise indices but I still have the following SQL which takes about a second to run
select hl7textinf0_.id as id332_, hl7textinf0_.accessionNum as accessio2_332_, hl7textinf0_.discipline as discipline332_, hl7textinf0_.filler_order_num as filler4_332_, hl7textinf0_.final_result_count as final5_332_, hl7textinf0_.first_name as first6_332_, hl7textinf0_.health_no as health7_332_, hl7textinf0_.lab_no as lab8_332_, hl7textinf0_.label as label332_, hl7textinf0_.last_name as last10_332_, hl7textinf0_.obr_date as obr11_332_, hl7textinf0_.priorSimply.ity as priority332_, hl7textinf0_.report_status as report13_332_, hl7textinf0_.requesting_client as requesting14_332_, hl7textinf0_.result_status as result15_332_, hl7textinf0_.sending_facility as sending16_332_, hl7textinf0_.sex as sex332_ from hl7TextInfo hl7textinf0_ where hl7textinf0_.accessionNum like ?

Java Melody Monitoring advises that its run in /lab/newLabUpload.do 
I don't care about a one second wait, and this is not your problem anyway
If you lack this monitoring tool you may activate the slow query log of 
mysql in /etc/mysql/my.cnf by adding the following
long_query_time=2
log-slow-queries=/var/log/mysql/log-slow-queries.log
log-queries-not-using-indexes

 Note that the slow-log itself can slow performance.  Turn it off when you are not using it!

5) A bigger faster box.  This is so rarely the issue that I hesitate to suggest it.  However if you are big enough (number of users say > 100) MySQL starts slowing down.  The real symptom of this is that people start volunteering for evening shifts so that they can get through their inboxes quickly.

You should only consider a bigger box(s) after you have exhausted the above.  In those cases there are ways to split off the OSCAR server(s) from multiple MySQL (or MariaDB) servers that load balance.  This is so over my head and experience that I shudder at contemplating this solution.


================
Peter Hutten-Czapski
Haileybury Ontario

"The attitude that ‘if rural people want these services they’ll have to come to the city to get them’ is simply not acceptable…” (Newbery, 1999)

Before printing, think about the environment. Avant d' imprimer, pensez à l'environnement.

On 19 October 2016 at 12:55, Daniel Ek <[hidden email]> wrote:
Can anyone provide advice on the problem of intermittently having issues with extremely slow  display of lab reports. These are HL-7 files I believe which are received from our hospital and from Lifelabs. They populate the Inbox of OSCAR (version 12.1) and when clicking to view, can take more than a minute to display, and at times the "system" will "time out" and fail to display. This happens whether one is accessing from the Lab file of the patient e-chart, or from our individual Inboxes.

This does not happen with Documents which are reports scanned into OSCAR.

There is some correlation with the time it takes to open and the length of the report, but frequently one is unable to open a simple lab test of one line only (ex. random glucose). I have had some success by clicking on other lab results in quick succession, and eventually one will open. And sometimes by returning to the original which did not open, it will open unexpectedly quickly.

I do not know anything about the server we currently use. Our OSP is Indivica, and they cannot define a specific reason for the problem, as it is not consistent.

Dan Ek
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Oscarmcmaster-ontario-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/oscarmcmaster-ontario-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Oscarmcmaster-ontario-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/oscarmcmaster-ontario-users