Generating JSON File performance problem

[menschmarkus]

Hi,

in the moment I am facing a timing problem generating a JSON File and offer it for downloading it ( ExportFile() ).
So what happens:
User has possibility to select a CSV File with e.g.1000 lines. Each line is checked and the app send a request via php to my server for database research. Reply comes back with a JSON String in an acceptable time (OK thats a server thing...). I select the JSON String in a variable and add each result which comes from my server. At last I create a file and write the string variable to it and export it. Now, it takes up to 2 minutes until the file is written and ready to be downloaded (System save file Window appears). To be honest, the file size was 60 MB and may be larger if I increase the number of lines in CSV file.
So 2 things happens I am not really happy about.
1. Preparation of file takes very long
2. Browser message appears an app takes too long for reaction: Stop or wait app.
Does someone have an idea for better handling of huge data?

Useful discussion start now

[Peter]

User has possibility to select a CSV File with e.g.1000 lines. Each line is checked and the app send a request via php to my server for database research.

Why not upload the whole file in one go and let the server do the work (including building the complete JSON)?

[menschmarkus]

Why not upload the whole file in one go and let the server do the work (including building the complete JSON)?

Hm,
I will try it...

[menschmarkus]

@Peter
facing a problem with uploading CSV file.
I tried falsams dropzone but this accept image files only. I checked the js file and found at line 342:

Code: Select all

acceptedFiles: null,

I changed it to

Code: Select all

acceptedFiles: text/csv,

(I also tried text/plain, text/comma-separated-values without success)
This did not work. The drop zone stays unaccesable so I could not select a file.

How can I release CSV Files for uploading, so where is the flag in js file?
(Sorry for this question but JS is not my world )

[Peter]

I tried falsams dropzone but this accept image files only.

I'm not so familiar with falsam's component. You can alternatively use my upload code: viewtopic.php?p=7249#p7249

Greetings ... Peter

[menschmarkus]

Thanks Peter, I will try your code.
I will come back and report....

[menschmarkus]

@ Peter: As promised, my report:

first of all thank you Peter, upload works fine.

Any Idea how to indicate the user about the progress status of Servers work?

[Peter]

Any Idea how to indicate the user about the progress status of Servers work?

Hard to say, because I don't know your project.

How big is the CSV you upload to the server? You could also send it to the server via HTTPRequest() The server will then send a kind of identification number back to the client. With this identification number the client could then ask the server cyclically how far the processing of the data has already progressed.

Greetings ... Peter

[menschmarkus]

I don't know you project

I just want to run Database request from a CSV File and send back a result (e.g. JSON string) of all requests in one reply.

How big is the CSV you upload to the server?

As I mentioned in my first post could be 1000 or up to 10000 lines

You could also send it to the server via HTTPRequest(). The server will then send a kind of identification number back to the client. With this identification number the client could then ask the server cyclically how far the processing of the data has already progressed.

OK inPB no problem but in SB? I do not have enough experience in SB to see the progress status actively.
In your sample for uploading files you finally upload it with a process called in HTTPRequest(). How long is this event alive, want to say how often can I fire a reply from my server to ctch the information? Debug mode shows after first return a status 200. I am a little confused about at what time a bind event stops listening.

rgds menschmarkus

[menschmarkus]

@Peter
OK, It took a moment but I found a suitable solution for me.
Unfortunately the result is too big. There is no value returned by the PHP script if the result exceed a size (I do not know exactly where it stops offering Data but it may be limited by PHP ini setting).
I will write the result to a file on my server which is no problem but how can I prepare it for downloading it from my server to the client and offer the download?

Do you have a similar simple JS script for SB like the one for uploading files?

rgds