Generating JSON File performance problem
-
- Posts: 37
- Joined: Thu Apr 10, 2014 3:35 pm
Generating JSON File performance problem
Hi,
in the moment I am facing a timing problem generating a JSON File and offer it for downloading it ( ExportFile() ).
So what happens:
User has possibility to select a CSV File with e.g.1000 lines. Each line is checked and the app send a request via php to my server for database research. Reply comes back with a JSON String in an acceptable time (OK thats a server thing...). I select the JSON String in a variable and add each result which comes from my server. At last I create a file and write the string variable to it and export it. Now, it takes up to 2 minutes until the file is written and ready to be downloaded (System save file Window appears). To be honest, the file size was 60 MB and may be larger if I increase the number of lines in CSV file.
So 2 things happens I am not really happy about.
1. Preparation of file takes very long
2. Browser message appears an app takes too long for reaction: Stop or wait app.
Does someone have an idea for better handling of huge data?
Useful discussion start now
in the moment I am facing a timing problem generating a JSON File and offer it for downloading it ( ExportFile() ).
So what happens:
User has possibility to select a CSV File with e.g.1000 lines. Each line is checked and the app send a request via php to my server for database research. Reply comes back with a JSON String in an acceptable time (OK thats a server thing...). I select the JSON String in a variable and add each result which comes from my server. At last I create a file and write the string variable to it and export it. Now, it takes up to 2 minutes until the file is written and ready to be downloaded (System save file Window appears). To be honest, the file size was 60 MB and may be larger if I increase the number of lines in CSV file.
So 2 things happens I am not really happy about.
1. Preparation of file takes very long
2. Browser message appears an app takes too long for reaction: Stop or wait app.
Does someone have an idea for better handling of huge data?
Useful discussion start now
as soon you do it right, it works !
Re: Generating JSON File performance problem
Why not upload the whole file in one go and let the server do the work (including building the complete JSON)?menschmarkus wrote:User has possibility to select a CSV File with e.g.1000 lines. Each line is checked and the app send a request via php to my server for database research.
-
- Posts: 37
- Joined: Thu Apr 10, 2014 3:35 pm
Re: Generating JSON File performance problem
Hm,Why not upload the whole file in one go and let the server do the work (including building the complete JSON)?
I will try it...
as soon you do it right, it works !
-
- Posts: 37
- Joined: Thu Apr 10, 2014 3:35 pm
Re: Generating JSON File performance problem
@Peter
facing a problem with uploading CSV file.
I tried falsams dropzone but this accept image files only. I checked the js file and found at line 342:
I changed it to (I also tried text/plain, text/comma-separated-values without success)
This did not work. The drop zone stays unaccesable so I could not select a file.
How can I release CSV Files for uploading, so where is the flag in js file?
(Sorry for this question but JS is not my world )
facing a problem with uploading CSV file.
I tried falsams dropzone but this accept image files only. I checked the js file and found at line 342:
Code: Select all
acceptedFiles: null,
Code: Select all
acceptedFiles: text/csv,
This did not work. The drop zone stays unaccesable so I could not select a file.
How can I release CSV Files for uploading, so where is the flag in js file?
(Sorry for this question but JS is not my world )
as soon you do it right, it works !
Re: Generating JSON File performance problem
I'm not so familiar with falsam's component. You can alternatively use my upload code: viewtopic.php?p=7249#p7249menschmarkus wrote:I tried falsams dropzone but this accept image files only.
Greetings ... Peter
-
- Posts: 37
- Joined: Thu Apr 10, 2014 3:35 pm
Re: Generating JSON File performance problem
Thanks Peter, I will try your code.
I will come back and report....
I will come back and report....
as soon you do it right, it works !
-
- Posts: 37
- Joined: Thu Apr 10, 2014 3:35 pm
Re: Generating JSON File performance problem
@ Peter: As promised, my report:
first of all thank you Peter, upload works fine.
Any Idea how to indicate the user about the progress status of Servers work?
first of all thank you Peter, upload works fine.
Any Idea how to indicate the user about the progress status of Servers work?
as soon you do it right, it works !
Re: Generating JSON File performance problem
Hard to say, because I don't know your project.menschmarkus wrote:Any Idea how to indicate the user about the progress status of Servers work?
How big is the CSV you upload to the server? You could also send it to the server via HTTPRequest() The server will then send a kind of identification number back to the client. With this identification number the client could then ask the server cyclically how far the processing of the data has already progressed.
Greetings ... Peter
-
- Posts: 37
- Joined: Thu Apr 10, 2014 3:35 pm
Re: Generating JSON File performance problem
I just want to run Database request from a CSV File and send back a result (e.g. JSON string) of all requests in one reply.Peter wrote:I don't know you project
As I mentioned in my first post could be 1000 or up to 10000 linesPeter wrote:How big is the CSV you upload to the server?
OK inPB no problem but in SB? I do not have enough experience in SB to see the progress status actively.Peter wrote: You could also send it to the server via HTTPRequest(). The server will then send a kind of identification number back to the client. With this identification number the client could then ask the server cyclically how far the processing of the data has already progressed.
In your sample for uploading files you finally upload it with a process called in HTTPRequest(). How long is this event alive, want to say how often can I fire a reply from my server to ctch the information? Debug mode shows after first return a status 200. I am a little confused about at what time a bind event stops listening.
rgds menschmarkus
as soon you do it right, it works !
-
- Posts: 37
- Joined: Thu Apr 10, 2014 3:35 pm
Re: Generating JSON File performance problem
@Peter
OK, It took a moment but I found a suitable solution for me.
Unfortunately the result is too big. There is no value returned by the PHP script if the result exceed a size (I do not know exactly where it stops offering Data but it may be limited by PHP ini setting).
I will write the result to a file on my server which is no problem but how can I prepare it for downloading it from my server to the client and offer the download?
Do you have a similar simple JS script for SB like the one for uploading files?
rgds
OK, It took a moment but I found a suitable solution for me.
Unfortunately the result is too big. There is no value returned by the PHP script if the result exceed a size (I do not know exactly where it stops offering Data but it may be limited by PHP ini setting).
I will write the result to a file on my server which is no problem but how can I prepare it for downloading it from my server to the client and offer the download?
Do you have a similar simple JS script for SB like the one for uploading files?
rgds
as soon you do it right, it works !