[Bug 162211] Copying to an external causes lots of missing files

[Markos Chandras]

http://bugs.kde.org/show_bug.cgi?id=162211


Markos Chandras hwoarang silverarrow gr changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |hwoarang-7SQBs15AKG+cDEF+Oi/***@public.gmane.org




--- Comment #7 from Markos Chandras <hwoarang silverarrow gr> 2008-10-13 17:14:22 ---
I can verify this problem. I am using kde-4.1.2 and when I try to copy data to
my usb flash ( kingstone data traveller 2GB ) which is formated in fat32 i get
nothing. The data seems to copy fine but when I plug the flash on another
computer the data is missing. The usb flash works fine when I do the same stuff
from kde3

[Richard Hartmann]

http://bugs.kde.org/show_bug.cgi?id=162211


Richard Hartmann richih-kde richih org changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |NEW
Ever Confirmed|0 |1




--- Comment #8 from Richard Hartmann <richih-kde richih org> 2008-10-13 17:32:10 ---
As this is now confirmed, any objection against giving it a higher priority?

[Richard Hartmann]

http://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #9 from Richard Hartmann <richih-kde richih org> 2008-10-13 17:34:51 ---
Markos: Can you provide as much information on how to reproduce this as
possible? Can you reproduce this? Does exactly the same thing happen ever time?

[Markos Chandras]

http://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #10 from Markos Chandras <hwoarang silverarrow gr> 2008-10-13 17:39:28 ---
Yes this happens every time with every usb storage device formated on fat32.
Havent tried ntsf or something else.

The way to reproduce it is quite simple ( for me at least ). Just plug a usb
device, try to copy a file using Konqueror-4 or dolphin. Then umount the usb
device and plug it again. The file is not there ...

[Richard Hartmann]

http://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #11 from Richard Hartmann <richih-kde richih org> 2008-10-13 17:49:32 ---
Does it unmount cleanly? Can you try to run

sync

before you detach?

[Gudy]

http://bugs.kde.org/show_bug.cgi?id=162211


Gudy jazzcommunication gmail com changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |jazzcommunication-***@public.gmane.org




--- Comment #12 from Gudy <jazzcommunication gmail com> 2008-10-17 19:12:39 ---
I confirm this problem, i got the same issue, with an EXTERNAL WD 500mb
harddisk formatted ext3 filesystem on the usb2.0....I was moving all my music
from the internal HD to the EXTERNAL HD, about 200GB the moving process was
smooth, no troubles. But when the next day i was seaching some music, i found
out that the first few files of a subdirectory were copied well (3..4 files)but
the remaining 10 files in that subdirectory had a lenght of 0 (zero)bytes
(average 50mb/file)
No error messages or whatever :(

maybe we need a verify option

[Markos Chandras]

http://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #13 from Markos Chandras <hwoarang silverarrow gr> 2008-10-17 21:02:15 ---
sync command before removing the disk seems to works most of the time but NOT
always. Still no error messages

[Richard Hartmann]

http://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #14 from Richard Hartmann <richih-kde richih org> 2008-10-19 14:00:09 ---
OK, I just copied some stuff to a thumb drive and thought I would try and
recreate the data loss, if possible (normally, I just use CLI).

Here's what I did

* Insert thumb drive
* Click the little plasma widget which informs me that a USB drive has been
connected, this opens Dolphin
* Copy stuff
* Click the volume on the left
* Select safe unmounting

I then ran mount on the CLI and lo, behold:

/dev/sda1 on /media/disk type vfat
(rw,nosuid,nodev,uhelper=hal,uid=1000,codepage=437,iocharset=utf8)


Dolphin does NOT complain that it could not unmount the drive! It just fails
and is silent about it. A user will, rightly, expect that it has been unmounted
cleanly.

Unmounting by hand (after moving to a different location within Dolphin as
Dolphin held the mount directory open) works just fine. No data loss.


Can anyone please confirm/deny that they acted in a similar way when their data
loss occured? If you can confirm, we found the issue. If you can't, I will open
a new bug.

[Markos Chandras]

http://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #16 from Markos Chandras <hwoarang silverarrow gr> 2008-10-19 20:41:07 ---
Yeah , indeed im waiting until the led turns off. Unfortunately i cant
re-produce it always :(

[Richard Hartmann]

http://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #15 from Richard Hartmann <richih-kde richih org> 2008-10-19 20:21:33 ---
Markos: When you sync manually, do you wait until the diode on your thumb drive
stops blinking? It's usually a good idea to wait for two seconds after you are
sure it has stopped, because, sometimes, it has not.

Please note that the sync command (same as umount, which calls sync(), as
well), may return normally, i.e. without error code when the hardware itself is
still busy.

[A.Mosteo]

http://bugs.kde.org/show_bug.cgi?id=162211


A Mosteo alejandro mosteo com changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |alejandro-hQBA2lZtZ6PQT0dZR+***@public.gmane.org

[Max]

https://bugs.kde.org/show_bug.cgi?id=162211


Max <knuckster-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |knuckster-***@public.gmane.org




--- Comment #17 from Max <knuckster gmail com> 2009-05-26 13:20:29 ---
I suppose copying itself is not the source of these problems. I've noticed that
copying to my flash mp3 player is a lot faster in KDE4, but it seems that data
is not actually written, but rather cached somewhere. So, when trying to umount
the drive through plasma applet or dolphin panel, I have to wait for a 1-3
minutes(!) while the data (~100MB usualy) is actually being written. The device
indicates writing process. Dolphin, waiting the umount, usualy gets timeout
from dbus daemon and shows a message in the bottom yellow panel.
IMO copying/caching routines to such devices should be reviewed.

[Pieter Vande Wyngaerde]

https://bugs.kde.org/show_bug.cgi?id=162211


Pieter Vande Wyngaerde <pietervandewyngaerde-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.
| |com




--- Comment #18 from Pieter Vande Wyngaerde <pietervandewyngaerde gmail com> 2009-06-16 11:22:11 ---
isn't there a "flush" mount option for removable media to write the stuff
directly ?

[Richard Hartmann]

https://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #19 from Richard Hartmann <richih-kde richih org> 2009-06-16 13:45:15 ---
Yes. Maybe it should be enabled by default..?

Problem is how do you determine when you want this or not? Especially with
external drives you actually work on, this would make things _really_ slow.

[a***@public.gmane.org]

https://bugs.kde.org/show_bug.cgi?id=162211


alexander-***@public.gmane.org changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |alexander-***@public.gmane.org




--- Comment #20 from <alexander schmiechen net> 2009-07-22 09:19:38 ---
I can confirm this and related problems with my Debian/SID installation (KDE
4.2.4). It seems to be a general problem of KDE4 handling files on a external
USB drive.

I just lost a lot of pictures from my camera's SD card connected to my desktop
PC via an USB reader. When trying to copy some files on my PC's hard drive I
got error messages. The resulting files were of correct size but contained no
information. Unfortunately, the original files were corrupted somehow during
the process...

[Richard Hartmann]

https://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #21 from Richard Hartmann <richih-kde richih org> 2009-07-22 10:27:57 ---
IMO, that either points to a general problem with the reader/SD
card/partition/formatting or KDE would do something _really_ stupid.

Did you m5 the files to see if corrupted original and copy are the same?

[Richard Hartmann]

https://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #22 from Richard Hartmann <richih-kde richih org> 2009-07-22 10:28:17 ---
md5, I mean..

[Gudy]

https://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #23 from Gudy <jazzcommunication gmail com> 2009-08-21 17:32:19 ---
please I wonder what is now the status of this serious bug, because its now a
year ago reported, and at that time I still got kubuntu 8.04 with kde 3.5.. now
2 distro's later its still there, I checked this out again on a new laptop &
desktop kubuntu 9.04 kde 4.2.. and tried to copy 50gb to my external harddisk ,
formatted ext3... still zerolength files without any message... You think you
backup your data safely

[Richard Hartmann]

https://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #24 from Richard Hartmann <richih-kde richih org> 2009-08-21 17:47:05 ---
Can you install 4.3, try to reproduce and tell us _exactly_ what you did?

Others: is there any sane way he can trace what is happening?

[FiNeX]

https://bugs.kde.org/show_bug.cgi?id=162211


FiNeX <finex-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |BS101286-***@public.gmane.org




--- Comment #25 from FiNeX <finex finex org> 2009-08-28 01:28:48 ---
*** Bug 205428 has been marked as a duplicate of this bug. ***

[Ben]

https://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #26 from Ben <BS101286 gmail com> 2009-08-28 01:38:23 ---
My bug was marked as a dup so here's the info from my own experience.

"I have a WD 1 tb external hard drive and I have twice now had almost all the
selected files deleted while moving from and to the external hard drive. Both
occasions involved more than 50,000 files(both times were photos), and I used
the "move to" command. The notification showed up but after a long time with
not activity it showed "completed moving x files" but the x number was only a
few hundred instead of the full selection. Since I used the "move to" command
there was no way to recover any of the files, undo move just moved the couple
hundred files back to the previous directory and the thousands of other files
disappeared as if they had never been.

running OpenSUSE 11.1 x64 with KDE 4.3 Factory"

also the 1 tb external is ntfs formatted with a mini usb connection and moving
the same files within my internal hard drive works fine.

[Paul]

https://bugs.kde.org/show_bug.cgi?id=162211


Paul <pablitobof-/***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |pablitobof-/***@public.gmane.org

[Patrick Stewart]

https://bugs.kde.org/show_bug.cgi?id=162211


Patrick Stewart <patstew-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |patstew-***@public.gmane.org




--- Comment #27 from Patrick Stewart <patstew gmail com> 2011-09-01 00:21:58 ---
This bug still seems to exist. I just copied ~60000 files to an external
(esata) backup drive, and hundreds of files weren't copied at all. The missing
files were completely random (eg one file in a photo/music album). Fortunately,
I happened to notice that some files were missing and recopied them all over
the top (skipping overwrites). It seems that all the files have transferred
now, but this is a very serious bug, and it's lucky I checked before deleting
the originals.
In case it is relevant, the files were copied by drag and drop in dolphin, and
both drives were ntfs and attached via sata.

[Gudy]

https://bugs.kde.org/show_bug.cgi?id=162211





--- Comment #28 from Gudy <jazzcommunication gmail com> 2011-09-01 01:27:17 ---
At one point i'm glad you confirmed my issue
this i reported already long long time ago (i guess 2008)
Its not related to ntfs, as in my case i used ext3 as format for the
external hd
When you use your external usb hd for home partition backup, all the time
you get useless backups when you check it

....i wonder when there will be some action taken :).....

regards
Jazz

Post by Max
https://bugs.kde.org/show_bug.cgi?id=162211
What |Removed |Added
----------------------------------------------------------------------------
--- Comment #27 from Patrick Stewart <patstew gmail com> 2011-09-01
00:21:58 ---
This bug still seems to exist. I just copied ~60000 files to an external
(esata) backup drive, and hundreds of files weren't copied at all. The
missing
files were completely random (eg one file in a photo/music album).
Fortunately,
I happened to notice that some files were missing and recopied them all
over
the top (skipping overwrites). It seems that all the files have
transferred
now, but this is a very serious bug, and it's lucky I checked before
deleting
the originals.
In case it is relevant, the files were copied by drag and drop in
dolphin, and
both drives were ntfs and attached via sata.

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=162211


Jaime Torres <jtamate-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |jtamate-***@public.gmane.org




--- Comment #29 from Jaime Torres <jtamate gmail com> 2011-09-27 10:42:09 ---
There is work in progress...

https://git.reviewboard.kde.org/r/102388/

[Mihkel Tõnnov]

https://bugs.kde.org/show_bug.cgi?id=162211

Mihkel Tõnnov <mihhkel-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |mihhkel-***@public.gmane.org

[David Edmundson]

https://bugs.kde.org/show_bug.cgi?id=162211

David Edmundson <kde-K9B6GWZOjNKs88QFB9wrT9Bc4/***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |dhvl2712-***@public.gmane.org

--- Comment #32 from David Edmundson <kde-K9B6GWZOjNKs88QFB9wrT9Bc4/***@public.gmane.org> ---
*** Bug 324606 has been marked as a duplicate of this bug. ***

[Jekyll Wu]

https://bugs.kde.org/show_bug.cgi?id=162211

Jekyll Wu <adaptee-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |adaptee-***@public.gmane.org

[Dhaval]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #33 from Dhaval <dhvl2712-***@public.gmane.org> ---
http://fpaste.org/37829/78572315/

This is my output from kdebugdialog when copying another batch of files. Some
folders are empty, however the number of empty folders is quite less than I
expected.

[Arthur Țițeică]

https://bugs.kde.org/show_bug.cgi?id=162211

Arthur Țițeică <arthur.titeica-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |arthur.titeica-***@public.gmane.org

[k***@public.gmane.org]

https://bugs.kde.org/show_bug.cgi?id=162211

kkoksvik-***@public.gmane.org changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |kkoksvik-***@public.gmane.org

--- Comment #34 from kkoksvik-***@public.gmane.org ---
I can confirm that copying large folder-structure between to external usb disks
using Dolphin , after copying finished many files where not copied an it looks
arbitrary witch files got copied or not. most folders came over(can not check
them all thousands of folders)
I am running Mint 14 KDE 4.11

[Dhaval]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #35 from Dhaval <dhvl2712-***@public.gmane.org> ---
It happened again, this time on a small folder. I was copying a folder with
about 10 large .mkv files from an External NTFS drive into a ext4 drive, and it
did not copy.

[Martin Sandsmark]

https://bugs.kde.org/show_bug.cgi?id=162211

Martin Sandsmark <martin.sandsmark-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|CONFIRMED |RESOLVED
CC| |martin.sandsmark-***@public.gmane.org
Resolution|--- |FIXED

--- Comment #36 from Martin Sandsmark <martin.sandsmark-***@public.gmane.org> ---
From reading the comments it seems like this is about Plasma/Dolphin/Solid not
reporting properly when it cannot unmount leading to people pulling out their
USB volumes prematurely, which now works for me (in 4.11, at least, probably
earlier as well); I get a warning message if the volume cannot be unmounted.

[Gudy]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #37 from Gudy <jazzcommunication-***@public.gmane.org> ---
Martin, this is NOT the case, we talk about usb connected harddisks, no
flash sticks,the harddisk is near permanently connected, the harddisk led
shows no activity for long long time, the system notifier also shows copy
done,its also not the disk cache "not flushed" as it goes about many
gigabytes missing much larger then the amount of ram and all programs
dolphin,nautilus,krusader......report copy is done.Next time when you
logon, many files zero length randomly
i think this issue is already since 2008


On Mon, Oct 7, 2013 at 3:48 PM, Martin Sandsmark

Post by Max
https://bugs.kde.org/show_bug.cgi?id=162211
What |Removed |Added
----------------------------------------------------------------------------
Status|CONFIRMED |RESOLVED
Resolution|--- |FIXED
From reading the comments it seems like this is about Plasma/Dolphin/Solid
not
reporting properly when it cannot unmount leading to people pulling out
their
USB volumes prematurely, which now works for me (in 4.11, at least,
probably
earlier as well); I get a warning message if the volume cannot be
unmounted.
--
You are on the CC list for the bug.

[Richard Hartmann]

https://bugs.kde.org/show_bug.cgi?id=162211

Richard Hartmann <richih-kde-KzTCLWY9SMsdnm+***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|FIXED |---

--- Comment #38 from Richard Hartmann <richih-kde-KzTCLWY9SMsdnm+***@public.gmane.org> ---
Having been involved with debugging in 2008(!) and pointing out the umount
issue myself, I agree that not all of these comments read that way.

In particular, https://bugs.kde.org/show_bug.cgi?id=162211#c34 specifically
states that 4.11 is still affected.

[Martin Sandsmark]

https://bugs.kde.org/show_bug.cgi?id=162211

Martin Sandsmark <martin.sandsmark-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |NEEDSINFO
Resolution|--- |WAITINGFORINFO

--- Comment #39 from Martin Sandsmark <martin.sandsmark-***@public.gmane.org> ---
(In reply to comment #37)

Post by Gudy
Martin, this is NOT the case, we talk about usb connected harddisks, no
flash sticks,the harddisk is near permanently connected, the harddisk led
shows no activity for long long time, the system notifier also shows copy
done,its also not the disk cache "not flushed" as it goes about many
gigabytes missing much larger then the amount of ram and all programs
dolphin,nautilus,krusader......report copy is done.Next time when you
logon, many files zero length randomly
i think this issue is already since 2008

Yes, but did you actually unmount it cleanly?

(In reply to comment #38)

Post by Gudy
Having been involved with debugging in 2008(!) and pointing out the umount
issue myself, I agree that not all of these comments read that way.
In particular, https://bugs.kde.org/show_bug.cgi?id=162211#c34 specifically
states that 4.11 is still affected.

Still could just be an unclean unmount in that comment.

Can someone definitely confirm that they _cleanly unmounted_ the drive in
question? Not just "I think I saw the safe to remove" message, but actually
checked with for example `mount`?

I went through all the code paths that are involved yesterday, and while I may
have missed something there (obviously) are no glaring errors in it. I have
also tested this myself with my own removable drives and with files of all
sizes and weird folder names, and I have been unable to reproduce it.

[k***@public.gmane.org]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #40 from kkoksvik-***@public.gmane.org ---
(In reply to comment #39)
Mint 14 KDE 4.11
I will try to explain with more details what I did when this happened.

Mounted two 1.3 TB USB2 NTFS drives using Dolphin with the intention of merging
the two so I could free up one of them for a different purpose.

Selecting all files on one drive i dragging them over to the other drive. The
copying takes a long time and when i return the copying process is done. I
immediately go looking for a file that is supposed to be copied over and can
not find it, looking at other files I recognize that many files are missing,
folders seams to have been copied OK.

Trying to reproduce this on smaller scales with only copying one folder with
files that did not copy the first time ends up with same result. Only way to
copy the files that did not get copied is to only copy one file at a time.

To me this seams like a problem with selection and iteration over the
selection(could be specific to ntfs)

I do not currently have access to the hardware that i used, but i will next
week, would be more than happy to help investigate.

[Max]

https://bugs.kde.org/show_bug.cgi?id=162211

Max <knuckster-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC|knuckster-***@public.gmane.org |

[Martin Sandsmark]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #41 from Martin Sandsmark <martin.sandsmark-***@public.gmane.org> ---
Could you try copying with the command line utility kioclient (kioclient cp
/src/ /target/)?

Also, could you share the full paths of the folders that didn't get copied, as
well as the target full path?

[Gudy]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #42 from Gudy <jazzcommunication-***@public.gmane.org> ---
I don't think it has to do with unmount, as by a normal system shutdown and
i not mean by pulling the plug, all drives are unmounted before the system
powers off itself.
But also by not logout, no screensaver or whatever to interfear with the
copy process it still happened
As it also seems to not makes any difference
what file system you use ntfs, ext2 ext3 ext4,or what application you
use..the issue is possibly deeper inside the kernel at the usb part, as
copying the same data to a NAS via ethernet 10/100, not wifi, has no
problems at all

I'm not a linux kernel expert, but i hope to be able to guide in what
direction they must search.

Its a real serious issue, as more and more people want to backup there
photo's ,movies,music and the "simple way" is get an external usb harddisk
to save the stuff and forget about it, knowing your data is there safe.....
to figure out later(hopefully the next day, and not next year that you
remember "oh yeah i had put it on my backup hd" )and that many files are
missing

Of course after the issue arrived , i never used a usb harddisk as backup
anymore
just put a new hd in the NAS (only on when i need it) copy the files to the
nas,and yes still verify it, it never failed
So there is a clear difference in how files are handled over ethernet and
usb

It is also not a usb harddisk power supply issue as i used a 12v 7ah lead
battery to power the usb hd (it had a 12v dc external power supply)
and used a shielded 20cm usb cable on a usb2 port

I will retry in one of the next weekends to copy by usb and post the
results including logs
Let me know what is the most usefull

best regards


On Tue, Oct 8, 2013 at 4:39 AM, Martin Sandsmark

Post by Max
https://bugs.kde.org/show_bug.cgi?id=162211
Could you try copying with the command line utility kioclient (kioclient cp
/src/ /target/)?
Also, could you share the full paths of the folders that didn't get
copied, as
well as the target full path?
--
You are on the CC list for the bug.

[f***@public.gmane.org]

https://bugs.kde.org/show_bug.cgi?id=162211

feniksa-***@public.gmane.org changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |feniksa-***@public.gmane.org

--- Comment #43 from feniksa-***@public.gmane.org ---
Confirm, that this bug still exists:

How to reproduce:
1. Insert any removable media
2. Start copy with dolphin any folder with a lot of files. I tried to copy
linux sources.
3. After some time i got such error:
Loading Image...

Additional information:
1. Media free space:
Filesystem Size Used Avail Use% Mounted on
/dev/sdc1 7.5G 1.2G 6.3G 16% /run/media/feniks/LIVE

2. Media information:
***@anna /mnt/webos_dev/linux $ sudo fdisk -l /dev/sdc

Disk /dev/sdc: 7963 MB, 7963803648 bytes, 15554304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc3072e18

Device Boot Start End Blocks Id System
/dev/sdc1 * 56 15554303 7777124 c W95 FAT32 (LBA)

3. Kdelibs version: 4.10.5-r1
4. Linux distro: gentoo
5. kernel: 3.10.7-gentoo-r1

[Baconmon]

https://bugs.kde.org/show_bug.cgi?id=162211

Baconmon <yasuna-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |yasuna-***@public.gmane.org

[Andi]

https://bugs.kde.org/show_bug.cgi?id=162211

Andi <agarthe-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |agarthe-***@public.gmane.org

--- Comment #44 from Andi <agarthe-***@public.gmane.org> ---
copy result seems to be predictable

I've got a folder on a NAS with about 2.000 files (10GB). Nearly each time the
folder is copied to the internal Notebook SSD the number of copied files is
different. kioclient used.

If you look at the copy icon in the task bar while a copy task is running you
can see how many files in total will be copied. Just click on the plus button
below the progress bar.
I made the experiance that in error case this value does not match with the
total number of files in the source folder. The result is an incomplete target
folder. But the target folder will always have exactly the number of files
shown in the task bar info while copying - the wrong result is predictable.

A second point I recognized is that if you repeat the copy procedure the result
will always be different in the number of copied files.

It seems that the file structure of the source folder is not read completely by
the KDE copy module.

The file names are quite long (2013-07-30 15.25.30 Canon EOS 650D.jpg) and the
NAS is not that fast (WD MyBook - ARM CPU). This might encourage the occurcance
of the error.

Using rsync to copy the folder works without errors on same setup. Performance
of NAS/network is about 30MB/sec reading for huge files. NAS is mounted as NFS.
fstab: 192.168.178.100:/nfs/Public /media/public nfs rw,noauto 0 0.

System:
Notebook i3/SSD
Debian Testing/KDE
NAS WDH1NC (ARM CPU)
FritzBox/Fixed IP

[Oleg]

https://bugs.kde.org/show_bug.cgi?id=162211

Oleg <sh-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |sh-***@public.gmane.org

--- Comment #45 from Oleg <sh-***@public.gmane.org> ---
I am having this issue when copying a few folders from my ext3 home directory
into USB flash drive or microSD (both Fat32) with Dolphin. I'm using Kubuntu
12.04 (all software up to date).

[Jason Straight]

https://bugs.kde.org/show_bug.cgi?id=162211

Jason Straight <j.straight-kde-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |j.straight-kde-***@public.gmane.org
| |t

--- Comment #46 from Jason Straight <j.straight-kde-***@public.gmane.org> ---
6 year old critical bug still exists. Just noticed it cost me most likely the
only existing baby photos of my son. Unfortunately, my backups [to three other
media] were made with KDE.

I'm currently sick over this.

A lot of good it did me to have 3 backups, when they were all incomplete. Maybe
if the bug isn't going to get fixed a warning should pop up.

[Jason Straight]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #47 from Jason Straight <j.straight-kde-***@public.gmane.org> ---
Okay, I can't imagine this would be hard to fix, after doing some more
investigating.

I have been playing with this and had similar results as Andi, plus noticed the
following:

Every time I copy the Pictures folder to anywhere, not just remote media, it
drops more than 1/2 the files with different numbers every time.

I've also found that simply right clicking the folder and choosing properties
will only give me a partial count, and every time I hit refresh - a different
total of files.

In total, this folder has 6677 files.

I think this has to do with the speed the query for files vs how long the kio
waits. And it has everything to do with the source, not the destination.

I have two external btrfs formatted HD's which [when checking properties in
KDE] show a count of about 3400 at first, but show the right total every time
afterwards hitting refresh in properties. After that count shows the actual
total of files, then copying them will copy them all.

I have one copy on an NTFS partition on my SSD [fuse - slow], which never shows
the right totals, no matter how many times I refresh. Anywhere between 2400 and
3000.

Another copy [mounted via cifs] on wireless only returns a count of about 1600
in properties.

Testing copying to various slow media such as flash drives and cifs mounts
works fine as long as the source media is fast enough to return the proper
total count.

[Gudy]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #48 from Gudy <jazzcommunication-***@public.gmane.org> ---
please don't think light about this issue, its not about the number of
files , but many files become zero length without any message about why it
failed. it looks like a kind of buffer overrun, to help imagine the issue
think about the old rs232 and you connect the rts and cts pins(hardware
handshake )(which is no problem IF the receiver is faster then the sender),
now rhe receiver is slower then the sender, the receive buffer becomes full
and the sender keeps on sending more, resulting in a loss of data
Of course with usb its all software implementation, i can't figure out if
the usb harddisk not sends a command to stop sending, or that the received
stop command is ignored,or delayed , just 1 byte missing from the file and
the checksum of the file is wrong ,possibly the harddisk controller puts
the file then at 0 length

Now the issue is why this goes wrong in linux without any message, in win
xp(since 2006 no win pc anymore,so i can't compare it with newer versions)
copying stops and you get frustrated why and waste of time , but at least
you know it wend wrong,better then thinking everything wend okay but when
you need your backup you have nothing or only a part

maybe this bug is deeply rooted in the kernel, as i always wondered why if
i make an SD card for the raspberry pi with the dd command it mostly result
in a not working card(rpi not boot)(even adjusting block sizes and whatever
possible), but when using windiskimager never any issue

This bug is serious (your most valuable emotional data lost ) and
difficult(extreme low level hardware and software knowledge needed, and
during the years many guys tried to close this bug report without doing
anything about it

Post by Max
https://bugs.kde.org/show_bug.cgi?id=162211
Okay, I can't imagine this would be hard to fix, after doing some more
investigating.
I have been playing with this and had similar results as Andi, plus
noticed the
Every time I copy the Pictures folder to anywhere, not just remote media,
it
drops more than 1/2 the files with different numbers every time.
I've also found that simply right clicking the folder and choosing
properties
will only give me a partial count, and every time I hit refresh - a
different
total of files.
In total, this folder has 6677 files.
I think this has to do with the speed the query for files vs how long the
kio
waits. And it has everything to do with the source, not the destination.
I have two external btrfs formatted HD's which [when checking properties in
KDE] show a count of about 3400 at first, but show the right total every
time
afterwards hitting refresh in properties. After that count shows the actual
total of files, then copying them will copy them all.
I have one copy on an NTFS partition on my SSD [fuse - slow], which never
shows
the right totals, no matter how many times I refresh. Anywhere between
2400 and
3000.
Another copy [mounted via cifs] on wireless only returns a count of about
1600
in properties.
Testing copying to various slow media such as flash drives and cifs mounts
works fine as long as the source media is fast enough to return the proper
total count.
--
You are on the CC list for the bug.

[Jason Straight]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #49 from Jason Straight <j.straight-kde-***@public.gmane.org> ---
I think we are actually talking about two different bugs here actually.

1. A bug where KDE (kioclient?) will not recognize all the files in a
folder/selection when it scans them prior to copying and when viewing
properties. This leads to you trying to copy or move a folder with 6700 files
and it actually only copies ~3400.

2. A different bug where you copy files and they exist on the destination, but
they are 0 bytes sized.

I have not experienced #2 myself.

[Jason Straight]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #51 from Jason Straight <j.straight-kde-***@public.gmane.org> ---
It's not "just" speed. There's more to it.

It gets me the right counts on the NTFS fuse filesystem for Program Files and
it takes a while to count them. For some reason the enumerating of files in my
Pictures folder kicks out before it's done.

Is there maybe a max time kioclient will wait while stating each file? And
perhaps one or more of my pictures takes a longer than usual to stat?

I'm open to try anything here. Give me some guidance on this. Hell, I'll even
set up a virtual machine with ssh and vnc access for the devs with access to my
photos.

I have kde debugging symbols installed, but my C skills are old and weak, and
my gdb skills are even less useful.

[Jason Straight]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #50 from Jason Straight <j.straight-kde-***@public.gmane.org> ---
Further evidence suggests this is a time issue. Also, a way to help reproduce
the issue.

On a USB3 mechanical drive formatted with btrfs filesystem I was not initially
experiencing fluctuating counts with kioclient properties on the folder.

When I put that same drive under a load using dd to write a file to it from
/dev/zero I started getting the problem where clicking refresh in properties
would yield random results on size and file count.

[Gudy]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #52 from Gudy <jazzcommunication-***@public.gmane.org> ---
Jason please not only look at the number of files, verify the size of the
directory ,source and destination in case there are zero length files, the
names are in the directory, but the total files size count is different
example , source =2000 files , total 6.3 GB, destination = 2000 fles total
4.8 GB
got the same issue by ext2, ext3,ext4,ntfs formatted external usb 2.0 hd
from western digital, and seagate

After that i not used usb harddisks anymore as the backup is unreliable,
but each time there is a new distro upgrade, i still try ,it to see if the
issue still exist , and from 2008 until now it has not changed......in a
few weeks after upgrading to 14.04 i will test again

Post by Max
https://bugs.kde.org/show_bug.cgi?id=162211
It's not "just" speed. There's more to it.
It gets me the right counts on the NTFS fuse filesystem for Program Files
and
it takes a while to count them. For some reason the enumerating of files
in my
Pictures folder kicks out before it's done.
Is there maybe a max time kioclient will wait while stating each file? And
perhaps one or more of my pictures takes a longer than usual to stat?
I'm open to try anything here. Give me some guidance on this. Hell, I'll
even
set up a virtual machine with ssh and vnc access for the devs with access
to my
photos.
I have kde debugging symbols installed, but my C skills are old and weak,
and
my gdb skills are even less useful.
--
You are on the CC list for the bug.

[Jason Straight]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #53 from Jason Straight <j.straight-kde-***@public.gmane.org> ---
I took my bug [as I think we're talking about two different ones] and created a
new one: https://bugs.kde.org/show_bug.cgi?id=333436

[d***@public.gmane.org]

https://bugs.kde.org/show_bug.cgi?id=162211

drtwox-***@public.gmane.org changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |drtwox-***@public.gmane.org

--- Comment #54 from drtwox-***@public.gmane.org ---
Using whatever the version of Dolphin is in Kubuntu 13.10, I copied 3701 files
in 1507 folders totaling 17.5GB to an external FAT32 drive, and Dolphin only
copied 777 files and 777 folders totaling 3GB. That 777 looks very suspicious.

[Arkadiusz]

https://bugs.kde.org/show_bug.cgi?id=162211

Arkadiusz <arekws-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |arekws-***@public.gmane.org

--- Comment #55 from Arkadiusz <arekws-***@public.gmane.org> ---
It's a pity that this still exists (5 years without fix?)
Few days ago I lost tons of family photos in KDE 4 (Ubuntu 14.04) when doing
move from internal drive to external drive (ctrl-x, ctrl-v). During that
process Dolphin has crashed. After that, I noticed, that all marked files on
internal drives were wiped out, but on external drive there was only some of
them (as proces was interrupted by crash)

It seems that Dolphin wipes out source files without waiting for confirmation
that files were successfuly moved to target location...

I think it should be treated as some kind of hot prio error. It is known since
2009, it can cause critical data loss.

[Gudy]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #56 from Gudy <jazzcommunication-***@public.gmane.org> ---
maybe we have to file this by mr Linus Torvalds self or at the kernel core
development group(please give me a hint how/where) as it seems this is not
the right place (or nobody has this specific knowledge)
....sad to see you used MOVE...so i can imagine the disaster..
.

Post by Max
https://bugs.kde.org/show_bug.cgi?id=162211
What |Removed |Added
----------------------------------------------------------------------------
It's a pity that this still exists (5 years without fix?)
Few days ago I lost tons of family photos in KDE 4 (Ubuntu 14.04) when
doing
move from internal drive to external drive (ctrl-x, ctrl-v). During that
process Dolphin has crashed. After that, I noticed, that all marked files
on
internal drives were wiped out, but on external drive there was only some
of
them (as proces was interrupted by crash)
It seems that Dolphin wipes out source files without waiting for
confirmation
that files were successfuly moved to target location...
I think it should be treated as some kind of hot prio error. It is known
since
2009, it can cause critical data loss.
--
You are on the CC list for the bug.

[Dawit Alemayehu]

https://bugs.kde.org/show_bug.cgi?id=162211

Dawit Alemayehu <adawit-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |imsnet-o+MxOtu4lMCHXe+***@public.gmane.org

--- Comment #57 from Dawit Alemayehu <adawit-***@public.gmane.org> ---
*** Bug 320890 has been marked as a duplicate of this bug. ***

[Mauro Molinari]

https://bugs.kde.org/show_bug.cgi?id=162211

Mauro Molinari <mauromol-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |mauromol-***@public.gmane.org

[Alex]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #58 from Alex <imsnet-o+MxOtu4lMCHXe+***@public.gmane.org> ---
as I wrote https://bugs.kde.org/show_bug.cgi?id=320890
bug reproduced on my PC.

and this https://bugs.kde.org/show_bug.cgi?id=333436 also reproduced.
perhaps there is a connection between them.

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #59 from Jaime Torres <jtamate-***@public.gmane.org> ---
Perhaps the bug is related to the use of non-reentrant methods in kio (as
signaled by cppcheck):

Non reentrant function 'readdir' called. For threadsafe applications it is
recommended to use the reentrant replacement function 'readdir_r'. (fixed in
KF5).
Non reentrant function 'strtok' called. For threadsafe applications it is
recommended to use the reentrant replacement function 'strtok_r'.
Non reentrant function 'getpwnam' called. For threadsafe applications it is
recommended to use the reentrant replacement function 'getpwnam_r'.
Non reentrant function 'getgrnam' called. For threadsafe applications it is
recommended to use the reentrant replacement function 'getgrnam_r'.
Non reentrant function 'getpwuid' called. For threadsafe applications it is
recommended to use the reentrant replacement function 'getpwuid_r'.
Non reentrant function 'getgrgid' called. For threadsafe applications it is
recommended to use the reentrant replacement function 'getgrgid_r'.
Non reentrant function 'getservbyname' called. For threadsafe applications it
is recommended to use the reentrant replacement function 'getservbyname_r'.
on reentrant function 'getprotobyname' called. For threadsafe applications it
is recommended to use the reentrant replacement function 'getprotobyname_r'.

[j***@public.gmane.org]

https://bugs.kde.org/show_bug.cgi?id=162211

justin.m.colley-***@public.gmane.org changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |justin.m.colley-***@public.gmane.org

[David]

https://bugs.kde.org/show_bug.cgi?id=162211

David <david.a.ordenes.d-***@public.gmane.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |david.a.ordenes.d-***@public.gmane.org

[d***@gmail.com]

https://bugs.kde.org/show_bug.cgi?id=162211

***@gmail.com changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

--- Comment #60 from ***@gmail.com ---
It happend just now on my machine

Ubuntu 12.10 +KDE plasma

From file manager nautlius , I copy four folders around 180GB from PC to USB
hardisk.

After two folders, around 110GB are copied, I count the file number on the usb
harddisk, there are same amount around 11k files as on my PC.

Then I stop file copying.

After that, from nautlius I try to copy the third folder over (around 7000+
files) to my USB harddisk.

Cancel transmition when there are 3000 files on the USB disk.

All looks good, counting files all, the same. So unmout, and remove USB disk.



When I open the usb harddisk on anther PC

There are only 2960 fiels lef t!!!!!!!!!!!!!!!!!!!!!!

Only a portion of first two folders.

The third folder and its subfolders DO NOT EXISTS at all.

[Baconmon]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #61 from Baconmon <***@gmail.com> ---
This bug just affected me!..
Actually I knew about this bug for a long time, but I stupidly forgot about it
when I was copying files for some one!..
I am in japan and about a year ago I hacked some japanese family's wii that I
am friends with.. I spent a long time downloading tons of wii games for them,
and also lots of NES and SNES roms and put all the games on to a hard-drive
attached to their wii..
This year they got a bigger HD for the games.. I stupidly forgot about this
bug, and used KDE to transfer tons of small files from their old HD to their
new HD..
It copied most (but not all) of the nes/snes roms, but only copied ONE wii
game!..

I had to tell them that all of their wii games are gone (save for 1) because
of a giant bug in linux.. Do you know how bad that makes linux look?..

This bug is the ONLY reason I can't recommend linux to any one.. Who would want
to use an OS that secretly erases your files when you transfer stuff via
USB!??..

Tons of people can easily reproduce this bug, so why hasn't it been fixed in
seven years??.. To me, this is by far the biggest bug in KDE.. I am just
wondering how many more years it is going to be until I can safely transfer
files using USB..

If this bug were in mac-os or windows, it would be fixed immediately, but since
it is linux, and people only fix things when they feel like it, a colossal bug
like this has still been unfixed for 7 years and counting..

I hope this bug is fixed this year.. I want to be able to talk about how great
linux is again.. But I just can't recommend it when it has a bug this huge in
it.. (And I am definitely not recommending gnome to any one though.. gnome
sucks..)

[juliaop]

https://bugs.kde.org/show_bug.cgi?id=162211

juliaop <***@yahoo.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@yahoo.com

--- Comment #62 from juliaop <***@yahoo.com> ---
use long path tool for easy solution.

[Christoph Feck]

https://bugs.kde.org/show_bug.cgi?id=162211

Christoph Feck <***@kde.org> changed:

What |Removed |Added
----------------------------------------------------------------------------

Post by Baconmon
Tons of people can easily reproduce this bug

But none of the developers. Also, it is still unclear if this issue is due to
removing the external device too early, i.e. before all the cached contents is
written out.

[Baconmon]

https://bugs.kde.org/show_bug.cgi?id=162211

But none of the developers. Also, it is still unclear if this issue is due to removing the external device too early, i.e. before all the cached contents is written out.

I didn't realize that none of the developers can manage to reproduce it, that
is really odd.. I think I am able to reproduce it on this computer, but not
sure if I would be able to give any information that would help?.. I know the
USB hard-drive had tons of small files in it (mainly every NES and SNES rom
ever made), so not sure if that triggers the bug to happen or not..

Also, for the record, I always unmount HDs/flash-drives/SD-cards using dolphin,
and I always wait until it says "you can unmount now", and actually before I do
that, I usually open a console and type "sync" and wait until that is done
before I unmount, because I am paranoid about the linux buffer saying files are
copied already, when actually they are being copied still in the background.. I
especially do this for important stuff..

Any way, I'm not sure if there is any information or tests or logs I can submit
that would help, but if there is, I would be happy to do it.. I use debian
stable, so I am stuck with this KDE version for a long time..

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=162211

Nathan <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

--- Comment #65 from Nathan <***@gmail.com> ---
I have a stereo that takes a usb thumb drive with mp3s or whatever loaded on
it. I just installed ubuntu and when i transferred some files to it everything
appeared to go well. When I put it in the stereo, tracks would play but either
get cut off randomly or even sometimes play one song for a few seconds, a part
of another song, back to the first and so on. I thought my files were corrupt,
but it plays fine on the computer. After reading this I wondered if I had made
the right choice installing ubuntu. I tried several things, none of which
worked, but I did learn something that might be helpful. I dropped the mp3
files into the root directory and they play fine now. if they are in folders,
despite the name, they always mess up. Hope this helps someone fix this. It's
really kinda annoying

[Austin S. Hemmelgarn]

https://bugs.kde.org/show_bug.cgi?id=162211

Austin S. Hemmelgarn <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

--- Comment #66 from Austin S. Hemmelgarn <***@gmail.com> ---
Reproduced using the following steps on Gentoo:
1. Created temporary LVM volumes for testing (test1 and test2) in a
thin-provisioning pool.
2. Formatted both with BTRFS.
3. Mounted at /mnt/test1 and /mnt/test2
4. Created large number (>65536) of files and folders in semi-random structure
on /mnt/test1 with very long path-name components (>200 bytes per level), and
filled files to random sizes with zero bytes.
5. Using cgroups, artificially limited disk bandwidth for the current shell
6. Using kioclient, attempted to copy files recursively form /mnt/test1 to
/mnt/test2.
7. Flushed all buffers to disk and unmounted /mnt/test2 (double sync followed
by umount from commandline).
8. Remounted /mnt/test2 and used diff to determine differences between
/mnt/test1 and /mnt/test2.
9. Repeated steps 6-8 multiple times, each time the results from diff were
different.
10. Repeated steps 6-8 using rsync instead of kioclient, and diff reported no
differences.
11. Repeated steps 6-8 without the disk bandwidth limited using cgroups, and
diff reported no differences.
12. Repeated steps 6-8 with shorter filenames and the disk bandwidth limited,
and still reproduced the issue.

Above results lead me to believe (without looking directly at the code) that
kio is timing out trying to gather directory contents and/or calling stat on
files.

[Austin S. Hemmelgarn]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #67 from Austin S. Hemmelgarn <***@gmail.com> ---
Reproduced using the following steps on Gentoo:
1. Created temporary LVM volumes for testing (test1 and test2) in a
thin-provisioning pool.
2. Formatted both with BTRFS.
3. Mounted at /mnt/test1 and /mnt/test2
4. Created large number (>65536) of files and folders in semi-random structure
on /mnt/test1 with very long path-name components (>200 bytes per level), and
filled files to random sizes with zero bytes.
5. Using cgroups, artificially limited disk bandwidth for the current shell
6. Using kioclient, attempted to copy files recursively form /mnt/test1 to
/mnt/test2.
7. Flushed all buffers to disk and unmounted /mnt/test2 (double sync followed
by umount from commandline).
8. Remounted /mnt/test2 and used diff to determine differences between
/mnt/test1 and /mnt/test2.
9. Repeated steps 6-8 multiple times, each time the results from diff were
different.
10. Repeated steps 6-8 using rsync instead of kioclient, and diff reported no
differences.
11. Repeated steps 6-8 without the disk bandwidth limited using cgroups, and
diff reported no differences.
12. Repeated steps 6-8 with shorter filenames and the disk bandwidth limited,
and still reproduced the issue.

Above results lead me to believe (without looking directly at the code) that
kio is timing out trying to gather directory contents and/or calling stat on
files.

[Austin S. Hemmelgarn]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #68 from Austin S. Hemmelgarn <***@gmail.com> ---
Apologies about the double comment.

[Alberto J. Ruiz]

https://bugs.kde.org/show_bug.cgi?id=162211

Alberto J. Ruiz <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

--- Comment #69 from Alberto J. Ruiz <***@gmail.com> ---
I'm an user that has slso been impacted by this bug.

Can you post output of kioclient with all debug options checked?

That is, open kdebugdialog, search kio in that window and check everything and
then running kioclient.

According to some comments here (
https://www.reddit.com/r/kde/comments/1ak1lz/psa_dolphin_exhibits_data_loss_do_not_use_it_for/
) outputing that info prevents the timing out in some way.

It might be useful to redirect the output like this:

kioclient copy src dst > errorlog.txt 2>&1

Also, thanks for your great effort.

By the way, this bug should be marked as CRITICAL.

[Austin S. Hemmelgarn]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #70 from Austin S. Hemmelgarn <***@gmail.com> ---
Yeah, but it might be a while. As of right now I don't have most of KDE
installed (out of the entirety of KDE suite, all I use personally is Okular,
Calligra, and a couple of the games). In fact I specifically installed
kioclient for the sole purpose of testing this (I'm trying to help get rid of
long standing bugs like this in FOSS projects whenever I stumble across them).
As I'm running Gentoo, this in turn means compiling everything, which based on
past experience with KDE will take a while. I will put it on the top of my
list for tomorrow however.

[Alberto J. Ruiz]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #71 from Alberto J. Ruiz <***@gmail.com> ---
(In reply to Austin S. Hemmelgarn from comment #67)

I can't reproduce the bug following your steps:


Kubuntu 14.04.2
Linux alberto-VirtualBox 3.16.0-46-generic #62~14.04.1-Ubuntu SMP Tue Aug 11
16:28:19 UTC 2015 i686 i686 i686 GNU/Linux

Enter root and then

1.
vgcreate -s 64M vg_thin /dev/XXXX

lvcreate -L 3G --thinpool tp_tecmint_pool vg_thin

lvcreate -V 1G --thin -n thin_vol_client1 vg_thin/tp_tecmint_pool
lvcreate -V 1G --thin -n thin_vol_client2 vg_thin/tp_tecmint_pool

2.
mkfs.btrfs /dev/vg_thin/thin_vol_client1
mkfs.btrfs /dev/vg_thin/thin_vol_client2


3.

mkdir /mnt/test1
mkdir /mnt/test2

mount /dev/vg_thin/thin_vol_client1 /mnt/test1
mount /dev/vg_thin/thin_vol_client2 /mnt/test2

4.
rm -rf /mnt/test1/*
./create_random_files.py /mnt/test1


##BEGINNING OF SCRIPT (Python3)
#!/usr/bin/env python3

import os
import random
import string
import sys

min_depth=3
max_depth=6

elem_by_folder_min=5
elem_by_folder_max=8

size_max_files_kb=50

letters_min=210
letters_max=220


files_and_folders_created=0

def rand_lett():
m=random.randint(letters_min,letters_max)
return ''.join((random.choice(string.ascii_letters) for _ in range(m)))


def create_file(name):
open(name,'wb').write(os.urandom(random.randint(1,size_max_files_kb*1024)))



def create_folder(folder_name,depth):
global files_and_folders_created

if depth != 0:
os.mkdir(folder_name)

os.chdir(folder_name)
for _ in range(random.randint(elem_by_folder_min,elem_by_folder_max)):
if depth < random.randint(min_depth,max_depth):
create_file(rand_lett())
create_folder(rand_lett(),depth+1)

files_and_folders_created=files_and_folders_created+2


os.chdir("..")


create_folder(sys.argv[1],0)
print ("Folders and files created:"+str(files_and_folders_created))

##END OF SCRIPT


5.

ls -ls /dev/vg_thin/*
Those are symlinks, following them:
ls -l /dev/dm-*

Major and minor numbers are:
252:4 y 252:5

grep blkio /proc/mounts || mkdir -p /cgroup/blkio ; mount -t cgroup -o blkio
none /cgroup/blkio
cgcreate -g blkio:/iothrottle

echo "252:4 5000000" > /cgroup/blkio/iothrottle/blkio.throttle.read_bps_device
echo "252:4 5000000" > /cgroup/blkio/iothrottle/blkio.throttle.write_bps_device

echo "252:5 5000000" >> /cgroup/blkio/iothrottle/blkio.throttle.read_bps_device
echo "252:5 5000000" >>
/cgroup/blkio/iothrottle/blkio.throttle.write_bps_device

5.1. Put every task in iothrottle group
ps axu | awk '{print $2}' | tail -n +2 | python3 -c 'import
sys;a=sys.stdin.readlines();
b=[open("/cgroup/blkio/iothrottle/tasks","a").write(val) for val in a]'


6.pre (I've doing all of that in root account)
chown -R XXX /mnt
logout


6.

rm errorlog.txt ; rm -rf /mnt/test2/* ; (kioclient copy /mnt/test1/*
/mnt/test2) >&errorlog.txt


7.

sync ; sync ; echo 3 | sudo tee /proc/sys/vm/drop_caches ; echo 3 | sudo tee
/proc/sys/vm/drop_caches ; sudo umount /mnt/test2

8.
Look for differences (Which files have been copied and their sizes):

sudo mount /dev/vg_thin/thin_vol_client2 /mnt/test2 && diff <(find /mnt/test1/
-printf '%P%s\n' | sort) <(find /mnt/test2/ -printf '%P%s\n' | sort)

Am I missing something?

[Austin S. Hemmelgarn]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #72 from Austin S. Hemmelgarn <***@gmail.com> ---
Interesting, I can't seem to reproduce it now either.
Guess I'm off to run hardware tests on my system.

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #73 from Nathan <***@gmail.com> ---
Would it help any if i used a different linux distribution? I am new to linux
and installed ubuntu

[j***@gmail.com]

https://bugs.kde.org/show_bug.cgi?id=162211

***@gmail.com changed:

What |Removed |Added
----------------------------------------------------------------------------
CC|***@gmail.com |

[Cacciatore]

https://bugs.kde.org/show_bug.cgi?id=162211

Cacciatore <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.co
| |m

--- Comment #74 from Cacciatore <***@gmail.com> ---
There s a lot of people out there having the exact same bug all these years and
it's really a pity that no action has been taken since the first reports came
out.
I recently fill a bug report https://bugs.kde.org/show_bug.cgi?id=352761
I'm really disappointed. Data loss is the most critical issue and after all
these years we are still afraid to use a simple command such as copy. Are we
being pushed in Windows?

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #75 from Jaime Torres <***@gmail.com> ---
Hello,

As you can see, if you read all the comments, this is a hard to reproduce
bug, even for the developers. For example, I've tried to reproduce it several
times, and I've seen it only once, and after doing the same steps, it didn't
reproduce again.
It is almost impossible to fix a bug if you can not reproduce it because you
have no way to know if it is fixed or not.

Has anyone that was able to reproduce this bug with KDE SC 4 tried to
reproduce it with the newest KF5 dolphin? (Just to know if it could still be
there or not)

[Cacciatore]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #76 from Cacciatore <***@gmail.com> ---
Hello Jaime, thank you for your answer.

As i state in the aformentioned bug report
(https://bugs.kde.org/show_bug.cgi?id=352761) this problem still exists in
these versions
kdelibs version is 4.14.11
dolphin 15.0.8.0
plasma-framework 5.13.0

I agree that you can't fix something that isn't reproduced by you (the
developers) and everyone clearly understand that. But, (and this is a big
emphazised ''but''), with a simple google search you can see thousands of
posts, all over the net, in various hardware setups, in various distros, and
all of them have experienced data loss when copy or moving things around. Some
of them can clearly provide more technical info that should give you insights
on whats happening.

We are willing to help you, thats why we raise all those bug reports, to
contribute, from our side, to the community, to bring more stable and compete
(and complete) de. We can realize that its a difficult task, to hunt down
those bugs, but the completion should be rewarding for all the community.
Personally i lack the developer skills, though i'm willing to help, to provide
the info you may want.

[w***@o2.pl]

https://bugs.kde.org/show_bug.cgi?id=162211

***@o2.pl changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@o2.pl

--- Comment #77 from ***@o2.pl ---
I've just copied 50 old dvd's with some backup files to an external usb drive
(using Krusader) and at the end I noticed that there are lots of missing files
on the usb drive. When I checked this out it turned out that not a single DVD
was completely copied to the usb drive! To be honest I was shocked that modern
version of OS may have such a bug. Moreover, from my point of view it is
trivial to reproduce the bug (as I said not a single dvd was properly copied),
hence I cannot believe that developers didn't noticed it.
My setup:
Ubuntu 14.04.3 LTS (with all updated installed)
krusader: 1:2.4.0~beta3-2
kdelibs-bin: 4:4.13.3-0ubuntu0.2
Now I am a bit scared how many times I didn't noticed that the 'copy' command
in krusader is not reliable.

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #78 from Jaime Torres <***@gmail.com> ---
Based on Alberto J. Ruiz script, I've made a similar script with infinite
repetition (until the copies are not equal to the original). It has been
running all week copying to an usb pendrive (I do not have an external usb hard
disk) and also copying to another sata harddisk. It should have copied more
than 10.000 files, and it has never reproduced this bug so far (using the
latest KF5 version compiled from sources last weekend).
But we will keep trying...

[w***@o2.pl]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #79 from ***@o2.pl ---
I made a simple screen-cast to show you the bug in action:
http://sendvid.com/dc92o1yu As I said, I have no problem with reproducing the
bug, so feel free to ask me for help in debugging an issue. Some additional
info: It seems that the destination of 'copy' doesn't matter - it is exactly
the same with usb and sata drives (both ext4). The source is ISO 9660. Each
time I press 'copy' on the folder the command works better - it finds more
files to copy. After 2 or 3 repeats it does recognize all files. After
unmounting/mounting everything starts from the begging.

[Alberto J. Ruiz]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #80 from Alberto J. Ruiz <***@gmail.com> ---
(In reply to wynajem_pn from comment #79)

Post by w***@o2.pl
http://sendvid.com/dc92o1yu As I said, I have no problem with reproducing
the bug, so feel free to ask me for help in debugging an issue. Some
additional info: It seems that the destination of 'copy' doesn't matter - it
is exactly the same with usb and sata drives (both ext4). The source is ISO
9660. Each time I press 'copy' on the folder the command works better - it
finds more files to copy. After 2 or 3 repeats it does recognize all files.
After unmounting/mounting everything starts from the begging.

All right. First, can you reproduce the bug using kioclient? "kioclient copy
source destination".

If so, open kdebugdialog. Tick Everyoption. Use kioclient to reproduce again.
If you can reproduce it with debug flags, paste the output of kioclient.

Thank you

[w***@o2.pl]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #81 from ***@o2.pl ---
Created attachment 94809
--> https://bugs.kde.org/attachment.cgi?id=94809&action=edit
output of kioclient copy

It is exactly the same with kioclient. Log file attached.

[Alberto J. Ruiz]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #82 from Alberto J. Ruiz <***@gmail.com> ---
Something looks fishy :

kioclient(12516)/kio (KIOJob) KIO::SimpleJob::~SimpleJob: Killing job
KIO::SimpleJob(0x17fe690) in destructor! "[
0: /usr/lib/libkdecore.so.5(kRealBacktrace(int)+0x3b) [0x7fb19cbe476b]
1: /usr/lib/libkio.so.5(KIO::SimpleJob::~SimpleJob()+0x86) [0x7fb19e393306]
2: /usr/lib/libkio.so.5(KIO::ListJob::~ListJob()+0x9) [0x7fb19e393609]
3: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QObject::event(QEvent*)+0x288)
[0x7fb19c51fc58]
4:
/usr/lib/x86_64-linux-gnu/libQtGui.so.4(QApplicationPrivate::notify_helper(QObject*,
QEvent*)+0x8c) [0x7fb19d174e2c]
5: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(QApplication::notify(QObject*,
QEvent*)+0x270) [0x7fb19d17b4a0]
6:
/usr/lib/x86_64-linux-gnu/libQtCore.so.4(QCoreApplication::notifyInternal(QObject*,
QEvent*)+0x6d) [0x7fb19c5074dd]
7:
/usr/lib/x86_64-linux-gnu/libQtCore.so.4(QCoreApplicationPrivate::sendPostedEvents(QObject*,
int, QThreadData*)+0x1ed) [0x7fb19c50ab3d]
8: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(+0x1aaf83) [0x7fb19c534f83]
9: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_dispatch+0x254)
[0x7fb198bd1e04]
10: /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x49048) [0x7fb198bd2048]
11: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_iteration+0x2c)
[0x7fb198bd20ec]
12:
/usr/lib/x86_64-linux-gnu/libQtCore.so.4(QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)+0x71)
[0x7fb19c5347a1]
13: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(+0x26bbe6) [0x7fb19d216be6]
14:
/usr/lib/x86_64-linux-gnu/libQtCore.so.4(QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)+0x2f)
[0x7fb19c5060af]
15:
/usr/lib/x86_64-linux-gnu/libQtCore.so.4(QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)+0x175)
[0x7fb19c5063a5]
16: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(QCoreApplication::exec()+0x89)
[0x7fb19c50bb79]
17: kioclient() [0x4046d8]
18: kioclient() [0x405518]
19: kioclient() [0x403a58]
20: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fb19bce2ec5]
21: kioclient() [0x403fa8]
]
"

Looks like a job is destroyed before finishing.



I almost forgot, sometimes dolphin enumerates files incorrectly in the
properties dialog (See bug 333436).
Have you lost files during this kioclient copy? I mean, you navigate through
folders and there aren't files that are supposed to be there. If that's the
case, you have successfully reproduced this bug after 7 years and we can start
working on it.

[w***@o2.pl]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #83 from ***@o2.pl ---
In fact during the logged run of 'kioclient copy' it copied 170MB out of 3,6GB
and there are more than 300 missing files.

[Cacciatore]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #84 from Cacciatore <***@gmail.com> ---
At last some progress. This is very promising and we hope you can find a
solution.
A little hint
This bug occurs even if you try to copy from an external disk (either ntfs or
ext4) to your internal disk. [the source is external-hdd and destination is
internal sata hdd]
It doesn't matter if dolphin enumerates files incorrectly in properties, the
actual problem is that it skips a tremendous amount of files to be copied.

[Alberto J. Ruiz]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #85 from Alberto J. Ruiz <***@gmail.com> ---
I'm not a KDE developer, only a rookie C++ developer.

In source code kdelibs-4.13.3/kio/kio/jobs.cpp :

void ListJob::slotResult( KJob * job )
{
if (job->error()) {
// If we can't list a subdir, the result is still ok
// This is why we override KCompositeJob::slotResult - to not set
// an error on parent job.
// Let's emit a signal about this though
emit subError(this, static_cast<KIO::ListJob*>(job));
}
removeSubjob(job);
if (!hasSubjobs())
emitResult();
}

Couldn't this be masking other errors not accounted?
We should check for ERR_ACCESS_DENIED and if it's not, mark the error the same
way KCompositeJob::slotResult does.
This is just a theory. The bug might be a completely unrelated.

[Alberto J. Ruiz]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #86 from Alberto J. Ruiz <***@gmail.com> ---
Created attachment 94949
--> https://bugs.kde.org/attachment.cgi?id=94949&action=edit
Possible patch for bug 162211

The faulty part of the copying process seems to be in ListJob.

1. ListJob fires the signal subError in case of error but it doesn't get
upwards. (Such as cannot enter subfolder).
2. The fact that the ListJob must work even if some subfolders cannot be
entered. ListJob::slotResult ommits all errors.
3. When such error happens, slotResults removes the subjob when it shouldn't.
It should return immediately like this function.

void FileCopyJob::slotResult( KJob *job)
{
.....
if ( job->error() )
{
.....
emitResult();
return;
}

....
}

I have reproduced the backtrace with "kioclient copy ~/src ~/dst"
~/src contains the following:
src:
total 4
drwxrwxr-x 3 alberto alberto 4096 oct 7 18:45 c

src/c:
total 548872
-rw-rw-r-- 1 alberto alberto 562036736 oct 7 14:01 a
drwx------ 2 root alberto 4096 oct 7 18:44 b

src/c/b:
total 1024
-rw-rw-r-- 1 alberto alberto 1048576 oct 7 18:44 aleatorio

The "a" file belongs to root and chmod'ed to 700 to emulate the "Cannot access
folder". The file is made of randomness.

After applying those changes, the backtrace has disappeared and kioclient does
not fail silently.

Tested on Kubuntu 14.04 . Patch for KdeLibs 4.14.12.

[Alberto J. Ruiz]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #87 from Alberto J. Ruiz <***@gmail.com> ---
Created attachment 94951
--> https://bugs.kde.org/attachment.cgi?id=94951&action=edit
Proposed patch

Actually, this is my proposed patch, not the otherone.

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=162211

***@free.fr changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@free.fr

--- Comment #88 from ***@free.fr ---
I'm affected by this bug too (well, not me directly, but my girlfriend, and as
a teacher it's very problematic for her). When we try to copy a folder that
contains several files and folders from her /home/Documents to a USB drive (no
matter what filesystem the drive is), KDE / Dolphin will copy the folders only.
Every folder will be there on the USB drive, but no file at all. We noticed
though, that sometimes if we copy the content of the upper folder (but not the
folder itself, just the content), it works.

She's running up-to-date Arch Linux (Plasma 5.4.2, latest frameworks). I
installed Thunar on her system and copying with Thunar works. So the problem
lies somewhere in KDE...

[Alex via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #89 from Alex <***@yandex.ru> ---
I lost the files when copying from hdd to hdd. Without using the USB drives.
Today checked on 2 PCs Archlinux in Konqueror, Krusader, Dolphin bug not
reproduced.
Previously, he consistently played. Now it is fixed.

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=162211

***@gmail.com changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

[Merike via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=162211

Merike <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

[Fernando Gutiérrez via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=162211

Fernando Gutiérrez <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

[Christoph Feck via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=162211

Christoph Feck <***@kde.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@live.com

--- Comment #90 from Christoph Feck <***@kde.org> ---
*** Bug 357875 has been marked as a duplicate of this bug. ***

[Me]

https://bugs.kde.org/show_bug.cgi?id=162211

Me <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

--- Comment #91 from Me <***@gmail.com> ---
I just lost a bunch of precious pictures due to a bug in Dolphin.

I was doing maintenance on my photo collection. I had previously downloaded
some jpg and raw files directly from the camera onto USB drive "A". I keep my
photos organized on USB drive "B". I thus attempted to move the photos from USB
drive A to USB drive B with Dolphin. Both USB A and USB B are multi GB SATA
3.5" hard drives with USB 3.0 interfaces. They are formatted EXT4.

As near as I can remember, here is what happened:

1) I opened Dolphin. I split the screen. I selected USB drive A on the left
panel. I selected USB drive B on the right panel.

2) I did a bit of browsing on USB A to ascertain which folders I wanted to
move. I created a new folder on USB drive B.

3) I selected about 10 folders containing several hundred images from drive A.
I dragged them over to the drive B panel in Dolphin and dropped them on the
folder I created. I selected Move from the pop up menu. Not copy.

4) The process started. I opened the KDE plasma notification window and
expanded the info to watch the files being copied. The process took about 10
minutes. The CPU load was very high, like 100%, on the plasma shell process.

5) At the conclusion of the process I just happened to open up a jpg to have a
look at it. I started scrolling through the files only to find that several sub
folders had images with file sizes of 0B. They could not be opened. The file
content was lost even if the handle was there.

6) When I went back to the source folders on drive A, the images are, of
course, gone.

The hard drives have both been fine. I ran fsck on them and both came up clean.

At no point in this process were the drives unmounted or disconnected. I got
called away from my desk and rechecked the files again later. The file sizes
were still zero.

I've seen files lost like this using Dolphin before. I suspect there is a bug
in the kio process when transferring large blocks of files to an external USB
drive.

What is interesting is that if you use mv or cp on the command line, the CPU
load is way, way lower than if the files are moved from Dolphin. Like 5% versus
100% and the command line operations are quite a bit faster.

$ uname -a
Linux localhost.localdomain 4.7.9-200.fc24.x86_64 #1 SMP Thu Oct 20 14:26:16
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

kf version 5.26.0.1.fc24

Been using Linux exclusively on my workstation since Redhat 8. I highly doubt
this is user error.

Let me know if you need more information.

[b***@kde.org]

https://bugs.kde.org/show_bug.cgi?id=162211

***@gmail.com changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

--- Comment #92 from ***@gmail.com ---
Is it fixed now as stated in this comment ?

https://www.reddit.com/r/linux/comments/8652zc/umm_gnome_shell_has_a_rather_big_memory_leak/dw2kgzy/

[Nate Graham]

https://bugs.kde.org/show_bug.cgi?id=162211

--- Comment #93 from Nate Graham <***@kde.org> ---
*** Bug 336624 has been marked as a duplicate of this bug. ***

[Dr. Chapatin]

https://bugs.kde.org/show_bug.cgi?id=162211

Dr. Chapatin <***@gmx.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmx.com

--- Comment #94 from Dr. Chapatin <***@gmx.com> ---
I think that I have been just affected by the problem reported here.
I extracted a zip file containing 104715 files and 1678 subfolders (900 MiB)
from a NTFS partition (internal hard disk) to my home (EXT4) using Ark.
I followed the steps below:

open dolphin
right click zip file, select "Extract > Extract archive to..."
browse to some place in another partition and create a new folder
click "extract" button to start extraction
when extraction is completed, open destination folder,
select all files/folders just extracted and press alt+enter to open properties
window
observe amount of files/sub-folders
open zip file using Ark, press alt+enter to open zip file properties,
observe "number of entries" info
Result: amount of files/sub-folders shown by Dolphin and "number of entries"
shown by Ark do NOT match

It happned twice. Each time the zip file was extracted, Dolphin shown a
different amount
of files/sub-folders in properties dialog. Now I can not reproduce anymore.

My system:
Arch Linux
plasma 5.12.5
frameworks 5.45
Dolphin 18.04