Re: [RndTbl] how to share/move sensitive data
Tar it up and compress it. Scp it. On Aug 24, 2016 17:19, "Trevor Cordes" <trevor@tecnopolis.ca> wrote: On 2016-08-24 Micah Garlich-Miller wrote:
- It's about 400 GB, but I dont know the data structure as I haven't seen it yet. ie, I'm not sure if its naturally chunked. - It's of a sensitivity that it needs to be crypted before being sent. - Sending on a crypted external harddrive is not acceptable in this situation.
I find a very easy, secure way to give people large files from my linux box is via apache. If you already have a linux box you run apache on then this is a breeze. If you don't, you need apache (or any other WS) and have it serve up port 80/443 to the external internet (either directly, through DMZ, or port forwarding). Since you want encrypted, SSL is a must, even if you put in a self-signed ssl cert (just send the fingerprint to the other user to verify in that case). What I do is I have a directory on my web server that has directory indexing turned off in apache. Then I put a directory in it that is some uberlong (like 64 chars) random alphanum string like pYwRYezIOz6EIAi4HijyF2hVe70L2V2hF3CR6UFMbvjCBgaI5XVsvDeiqzTJa307 Then in that directory have directory indexing turned on (you can use .htaccess for this). Put your big files in there. Then email or otherwise get the link to your end user: https://foo.com/hidden/pYwRYezIOz6EIAi4HijyF2hVe70L2V 2hF3CR6UFMbvjCBgaI5XVsvDeiqzTJa307/ Voila, secure, encrypted access to the files. For someone else on the net to gain access, they'd have to intercept your email/backchannel with the link, or guess the random string, which is basically impossible for this 62**64 string (or have local shell access to your server, though creative use of apache group permissions can mitigate this). If you wanted to you could even add .htpassword basic auth to the above. (Also, make sure you don't provide an un-SSL port 80 http:// link to it!) Test that your no-dirindex is indeed working by going to: https://foo.com/hidden/ which should give you an error saying indexing denied. As for breaking up the files, you could use split to break them into chunks, which might be prudent, though many http downloaders/browsers will resume a broken connection so even a single 400G file might be ok (try using wget with resume options for it). To split you'd do: split -b 1000000000 infile dl-me for 1GB-ish chunks. On the other side it's just: cat dl-me* > original-file Even Windbloze might be able to do it, maybe with type? type dl-me*.* > original-file ??? not sure on that one, maybe powershell gives us better options now. Final note, this transfer will saturate your modem's upload connection, so you might want to schedule it for a weekend or something. Ratelimiting with apache is kind of a pain, but the receiving end could use wget's rate-limiting options (aim for 75% your nominal u/l bandwidth maybe). Or you can setup qos / ratelimiting (using tc and iptables) for egress web traffic on your linux web server or linux router you control, but that gets complicated if you've never done it before. Final final note, you may tick off your ISP and/or exceed your monthly transfer limits (uploads are usually quite limited vs downloads). 400GB is a huge amount to upload for normal home/SMB shaw/mts accounts. (You could bzip (or better) the file before starting the whole process to make it smaller, if both sides have enough disk space for 2 copies.) _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
On 2016-08-24 Robert Keizer wrote:
Tar it up and compress it. Scp it.
Hah, a bit simpler than my way. But do keep in mind scp won't resume and if the sender is on a 2Mb up MTS DSL connection or similar it could take a looooong time to upload and no guarantee either side's connection or computers wouldn't go wonky during that time. I'm not sure I did the math right, but I calculate at 2Mb/s 400GB would take 18 *days*??!? scp would be good if you chunked the pieces real small, but bad for a single 400GB file, unless both sides have a great, fast, reliable pipe. You've been spoiled by fast pipes, Rob!! P.S. Maybe sneakernet would be a better option if your pipe is small... Buy 4 128GB SD cards or 1 500GB SSD drive, use split and gpg -c to chunk and encrypt; throw in a Xpresspost or UPS Express envelope and ship 1-2 day for around $15-$30. Figure 1 day to write the data (especially on slow SD cards, SSD would be waaay faster), 1-2 days en route. All done in 3 days. But you'll spend ~$250 on the cards/ssd, but you can keep it for use another day.
If network transfer is an option, tools like rsync or Unison over SSH are your friend. +1 for physical media for bandwidth though. Tim
On Aug 25, 2016, at 2:37 AM, Trevor Cordes <trevor@tecnopolis.ca> wrote:
On 2016-08-24 Robert Keizer wrote: Tar it up and compress it. Scp it.
Hah, a bit simpler than my way.
But do keep in mind scp won't resume and if the sender is on a 2Mb up MTS DSL connection or similar it could take a looooong time to upload and no guarantee either side's connection or computers wouldn't go wonky during that time.
I'm not sure I did the math right, but I calculate at 2Mb/s 400GB would take 18 *days*??!?
scp would be good if you chunked the pieces real small, but bad for a single 400GB file, unless both sides have a great, fast, reliable pipe.
You've been spoiled by fast pipes, Rob!!
P.S. Maybe sneakernet would be a better option if your pipe is small... Buy 4 128GB SD cards or 1 500GB SSD drive, use split and gpg -c to chunk and encrypt; throw in a Xpresspost or UPS Express envelope and ship 1-2 day for around $15-$30. Figure 1 day to write the data (especially on slow SD cards, SSD would be waaay faster), 1-2 days en route. All done in 3 days. But you'll spend ~$250 on the cards/ssd, but you can keep it for use another day. _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Hi all, I appreciate all the comments and thoughts, thank you. I'm just going to sum up what happened even though its a bit of a mess. Basically, the requirements that were given around privacy and for it being a non-physical approach evaporated. The recipient of the data eventually came back and said the network in that part of the world is incapable of receiving large files, so please mail physical. So thats whats going to be done, a crypted external hard-drive. I am grateful that this is the solution was deemed acceptable because in this particular case it seemed the best. The approach that we were preparing to do was to crypt and chunk using 7-zip which looks like a great product for exactly this types of problem. We were then going to make it available via sftp to the recipient. The reason for setting it up that way was so the recipient could take advantage of the faster download speeds (as compared to upload). Thanks again, Micah On Thu, Aug 25, 2016 at 9:15 AM, Tim Lavoie <tim@fractaldragon.net> wrote:
If network transfer is an option, tools like rsync or Unison over SSH are your friend. +1 for physical media for bandwidth though.
Tim
On Aug 25, 2016, at 2:37 AM, Trevor Cordes <trevor@tecnopolis.ca> wrote:
On 2016-08-24 Robert Keizer wrote: Tar it up and compress it. Scp it.
Hah, a bit simpler than my way.
But do keep in mind scp won't resume and if the sender is on a 2Mb up MTS DSL connection or similar it could take a looooong time to upload and no guarantee either side's connection or computers wouldn't go wonky during that time.
I'm not sure I did the math right, but I calculate at 2Mb/s 400GB would take 18 *days*??!?
scp would be good if you chunked the pieces real small, but bad for a single 400GB file, unless both sides have a great, fast, reliable pipe.
You've been spoiled by fast pipes, Rob!!
P.S. Maybe sneakernet would be a better option if your pipe is small... Buy 4 128GB SD cards or 1 500GB SSD drive, use split and gpg -c to chunk and encrypt; throw in a Xpresspost or UPS Express envelope and ship 1-2 day for around $15-$30. Figure 1 day to write the data (especially on slow SD cards, SSD would be waaay faster), 1-2 days en route. All done in 3 days. But you'll spend ~$250 on the cards/ssd, but you can keep it for use another day. _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
-- Micah Garlich-Miller <micahgarlichmiller@gmail.com>
Would strongly suggest GPG / PGP ( https://en.wikipedia.org/wiki/GNU_Privacy_Guard, https://en.wikipedia.org/wiki/Pretty_Good_Privacy ) instead of 7zip. On Fri, Aug 26, 2016 at 11:06 AM, Micah Garlich-Miller <micahgarlichmiller@gmail.com> wrote:
Hi all,
I appreciate all the comments and thoughts, thank you.
I'm just going to sum up what happened even though its a bit of a mess.
Basically, the requirements that were given around privacy and for it being a non-physical approach evaporated. The recipient of the data eventually came back and said the network in that part of the world is incapable of receiving large files, so please mail physical. So thats whats going to be done, a crypted external hard-drive. I am grateful that this is the solution was deemed acceptable because in this particular case it seemed the best.
The approach that we were preparing to do was to crypt and chunk using 7-zip which looks like a great product for exactly this types of problem. We were then going to make it available via sftp to the recipient. The reason for setting it up that way was so the recipient could take advantage of the faster download speeds (as compared to upload).
Thanks again, Micah
On Thu, Aug 25, 2016 at 9:15 AM, Tim Lavoie <tim@fractaldragon.net> wrote:
If network transfer is an option, tools like rsync or Unison over SSH are your friend. +1 for physical media for bandwidth though.
Tim
On Aug 25, 2016, at 2:37 AM, Trevor Cordes <trevor@tecnopolis.ca> wrote:
On 2016-08-24 Robert Keizer wrote: Tar it up and compress it. Scp it.
Hah, a bit simpler than my way.
But do keep in mind scp won't resume and if the sender is on a 2Mb up MTS DSL connection or similar it could take a looooong time to upload and no guarantee either side's connection or computers wouldn't go wonky during that time.
I'm not sure I did the math right, but I calculate at 2Mb/s 400GB would take 18 *days*??!?
scp would be good if you chunked the pieces real small, but bad for a single 400GB file, unless both sides have a great, fast, reliable pipe.
You've been spoiled by fast pipes, Rob!!
P.S. Maybe sneakernet would be a better option if your pipe is small... Buy 4 128GB SD cards or 1 500GB SSD drive, use split and gpg -c to chunk and encrypt; throw in a Xpresspost or UPS Express envelope and ship 1-2 day for around $15-$30. Figure 1 day to write the data (especially on slow SD cards, SSD would be waaay faster), 1-2 days en route. All done in 3 days. But you'll spend ~$250 on the cards/ssd, but you can keep it for use another day. _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
-- Micah Garlich-Miller <micahgarlichmiller@gmail.com>
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
participants (4)
-
Micah Garlich-Miller -
Robert Keizer -
Tim Lavoie -
Trevor Cordes