I worked on processing some large volumes of files with python and used a for loop for processing (reading, calculating and plotting) at each iteration, and then save the figures to my computer as well. IPython is used when I run the code, and matplotlib is in the interative mode and displays no the figures (ie., without show() in the code). The backend default is GTKAgg.
A simple problem that I ran into is: the memory of the computer was used more and more as the iteration number increased. This can bring the risk of crashing the system if there are a lot of files to be processed. Using a profiler from python, it was shown that the built-in method "write_png" took the longest portion of time than other functions. I realised that I did not clear the figures in each iteration and this causes the memory problem ! So I add a line at the end of an iteration ie., clf(), in order to clear all figures of each iteration (note: clf() is better to be used here than close(), because we do not use show() to display the figures and there is no reason to close a figure which is not displayed).
This indeed solve the memory problem and the code can be run quickly, and in the end of run we can exit ipython gracefully. The same code can be run without problem using python as well.
Aucun commentaire:
Enregistrer un commentaire