8bitOrange

Nerd Alert: Web Workers and the Filesystem

So I’m building something right now in pure Javascript. No PHP, it allows you to upload images, briefly edit them (basic things like brightness, size, crop, etc), and then continue on to the web service that I am back ending it with. 

I thought that it would be great if on image upload, it backed up the file to the filesystem. This was going well until I uploaded a lot of pictures at the same time, things would get hairy and eventually bomb out. Then a flash bulb went off and I decided to use Web Workers to upload the image, since workers have access to the file system and the image could continue to get processed in the meantime, I thought this would be a great way to do it. I started having problems though, so I’m going to list this stuff out here, if any of the 3 of you who read this and would know this stuff have a solution I’m open to trying anything, but at this time I have moved on without either tech.

1. Debugging workers: It’s a pain, at this time there seems to be no real solution around it, you just have to post messages back and hope they make it. There is a lot of finger crossing when it comes to web workers and a lot of try/catch to see if you can log an error before everything dies
2. Silently dying workers: It was really weird, if I uploaded 5 at a time everything would go great and they would load as expected. If I uploaded a lot, and I’m talking a 50-100, they would load in anywhere from 7-12 and then hang. Most of the time dying error free without ever finishing their job. I’m assuming the bulk (7-12 at a time) loading was due to available threads, but could not figure out a way to manage this due to the lack of debugging. I tried storing all active workers by reference in an array and then closing off the ones that finished, but that didn’t seem to help ( I am assuming they are doing some trash management of their own ).
3. File names…oh file names. You are the bastard process that screws up everything. I noticed that some files were bombing out and others weren’t. There, at first, was seemingly no pattern as to why one image would load and another wouldn’t. Then I realized it was all in the name. The ones that were dying had a “#” in them. I couldn’t find anything about name issues, but it appears that #’s are not allowed. So that’s all fine and dandy, just change the name right? Well I don’t have access to the file object in any meaningful way to change it until its uploaded (then I could do ye ol “moveTo” and change the name). I tried just manually changing it, but that wouldn’t work, which leaves me with the only other solution, which is to load each image into a canvas, output a data/image and save that to the filesystem with an arbitrary name. That is way to much “pre” work for something that was going to be a convenience for me. So I moved on without.

These three issues had me sidetracked for 2 days trying to get them to work, I ultimately decided to forgo it. Now instead, when you click to edit an image I will save a copy to my file system and version it off, that way you can go back to “snapshots” of the photo during editing, otherwise, for now, it is all destructible (don’t hit refresh).

Anyone else tried to tackle these beasts and run into these issues? Have any work arounds?