helprelax - Support: sr #3118, Optimization for faster reading of...

 
 
Show feedback again

sr #3118: Optimization for faster reading of state files

Submitted by:  Troels E. Linnet <tlinnet>
Submitted on:  Fri 14 Feb 2014 09:25:22 AM UTC  
 
Category: NonePriority: 5 - Normal
Severity: 1 - WishStatus: Postponed
Assigned to: NoneOpen/Closed: Closed
Operating System: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 17 Jun 2014 06:40:02 PM UTC, comment #1:

For large amount of pipes and data, just save these as individual states.

Troels E. Linnet <tlinnet>
Project Member
Fri 14 Feb 2014 09:25:22 AM UTC, original submission:

This follows the discussion in:
http://thread.gmane.org/gmane.science.nmr.relax.devel/4999

I am storing different runs of R2eff values into state files.
Their file size is app. 5 MB each.
The loading time is about 20-30 seconds for a state file.

After turning of comprepression, the file size reached ~70 MB.
But with no considerable increase in reading speed.

The optimization should probably be found here:

----------
Converting between data structures in the relax data store and the XML. These are all the to_xml() and from_xml() functions you can find in the 'data_store' package.

There have been zero optimisations here and, in some parts, the code is likely to be extremely inefficient (too much looping, inefficient ordering of loop nesting, etc.).
This would be the best target for optimisations.
And the loops are where you will squeeze out the most.
To find the relevant functions/methods, just run:

$ grep -r "def to_xml\|def from_xml" . --exclude-dir=".svn"

To perform any optimisations, the test suite must be used to make sure that nothing breaks.
If you notice a breakage in a save file you are using as a test for yourself while the test suite passes, then a new
system test needs to be created using a heavily truncated save file.

Troels E. Linnet <tlinnet>
Project Member

 

(Note: upload size limit is set to 1024 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tlinnet (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 17 Jun 2014 06:40:02 PM UTCtlinnetOpen/ClosedOpen=>Closed
    Show feedback again

    Back to the top


    Powered by Savane 3.1-cleanup