MySQL Option

[Petahhh]

Sqlite does not seem powerful enough to handle uploading of directories with 20+ million files. An option to use a MySQL database, perhaps as a flag, will be helpful for users with this resources and this need.

[cudevmaxwell]

I wouldn't agree, http://www.sqlite.org/limits.html. Even if every file path entry is 10 KB on disk (very long paths!) the DB would be 200GB, which is well under the limit for SQLite.

However, if there's interest, I can replace the raw SQL calls in my PR and instead use sqlalchemy. Then anyone could use any db they wanted.

[Petahhh]

Thanks for the feedback. The option to use any database you want would be really convenient.

Currently, I've been unable to upload a directory with 17 million files. I repeatedly get a "Database Error: disk I/O error."

[cudevmaxwell]

How much free space is on the drive?

[Petahhh]

About 200GB

[cudevmaxwell]

I'll try to reproduce.

Edit: Creating many, many small files.

Edit 2: At 2 of 20 million.

Edit 3: At 9 of 20 million.

[cudevmaxwell]

Created over 22 million test objects, here's some numbers:

$ time swiftbulkuploader --db=/tmp/test.db prepare ~/development/test_manyfiles/

real    48m7.618s
user    13m54.523s
sys     19m42.090s

$ time swiftbulkuploader --db=/tmp/test.db count
Stored accessible file paths: 22159415
  Not Uploaded: 22159415
      Uploaded: 0
  Upload Error: 0
Inaccessible file paths: 0
Inaccessible directory paths: 0

real    1m47.469s
user    0m25.491s
sys     0m24.902s

So, very sluggish. I'll run the upload command over the weekend. Good test of the tool, and also a good test of the beta cluster.

[cudevmaxwell]

Now that I think about it, is > 22 million inserts in 50 minutes that slow? That's over 7000 inserts per second.

[cudevmaxwell]

@Petahhh, did the error come up in the prepare step, or the upload step?

[cudevmaxwell]

Stopping the upload command now. Got to ~140,000 files in 5 hours, which is too slow. No errors from SQLite, though.