Hullo all.

I have a hard disk, 250Gb or so, in an external USB 2.0 HDD enclosure.
The disk is now full. Specifically, it is full of old backups. Each
backup is in its own directory, and there are about 45 directories at
the top level. Each directory contains about 300,000 files and
directories, mostly very small. The files are pretty evenly distributed
through the directories and subdirectories. The disk is an ext3
filesystem.

I now want to recycle the drive and start filling it up with new
backups. So I have started deleting the old backups. I don't want to
delete ALL the backups - just the oldest ones - so I can't just reformat
the disk.

Problem is, it is taking literally hours to remove each backup
directory. Why is this so? Is there any faster way than "rm -fr" to
remove a directory? Should I use a different filesystem type next time?

The only wrinkle here is that the backup directories were created using
rsynch, hard linking to identical files rather than copying them anew.
But that is a simple link count - I can't see it adding *this* much pain
to a simple deletion!

Clues?

Regards, K.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Karl Auer (kauer*******/~kauer/                   +61-428-957160 (mob)

GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF

I would rsynch the files you want to any other hard drive and then just
format the one you want to reuse.

--

    Karl F. Larsen, AKA K5DI
    Linux User
    #450462    http://counter.li.org.
         Key ID = 3951B48D

[snipped Karls advice to copy everything to a new drive and format the
original - not entirely sure what this was supposed to achieve other
than moving data unnecessarily]

You could mount your drive with the "data=writeback" option for the
duration of your deletion session. Data consistency in the event of a
power failure will be similar to an ext2 filesystem as you're now
running unjournalised.

If that makes you nervous, switch to data=ordered. Faster than
journalised but without the risk of completely switching journaling off.

Just don't forget to remount the drive with data=journal before putting
the drive back into use.

Additionally, you can increase the commit value (commit=5 is the
default, measured in seconds). Increase it to improve throughput at the
expense of recoverability.

I would recommend you familiarise yourself with the "ext3.txt" file...
I'll post it here just in case.

Ext3 Filesystem
===============

Ext3 was originally released in September 1999. Written by Stephen
Tweedie
for the 2.2 branch, and ported to 2.4 kernels by Peter Braam, Andreas
Dilger,
Andrew Morton, Alexander Viro, Ted Ts'o and Stephen Tweedie.

Ext3 is the ext2 filesystem enhanced with journalling capabilities.

Options
=======

When mounting an ext3 filesystem, the following option are accepted:
(*) == default

journal=update Update the ext3 file system's journal to the current
format.

journal=inum When a journal already exists, this option is ignored.
Otherwise, it specifies the number of the inode which
will represent the ext3 file system's journal file.

journal_dev=devnum When the external journal device's major/minor
numbers
have changed, this option allows the user to specify
the new journal location.  The journal device is
identified through its new major/minor numbers encoded
in devnum.

noload Don't load the journal on mounting.

data=journal All data are committed into the journal prior to being
written into the main file system.

data=ordered (*) All data are forced directly out to the main file
system prior to its metadata being committed to the
journal.

data=writeback Data ordering is not preserved, data may be written
into the main file system after its metadata has been
committed to the journal.

commit=nrsec (*) Ext3 can be told to sync all its data and metadata
every 'nrsec' seconds. The default value is 5 seconds.
This means that if you lose your power, you will lose
as much as the latest 5 seconds of work (your
filesystem will not be damaged though, thanks to the
journaling).  This default value (or any low value)
will hurt performance, but it's good for data-safety.
Setting it to 0 will have the same effect as leaving
it at the default (5 seconds).
Setting it to very large values will improve
performance.

barrier=1 This enables/disables barriers.  barrier=0 disables
it, barrier=1 enables it.

orlov (*) This enables the new Orlov block allocator. It is
enabled by default.

oldalloc This disables the Orlov block allocator and enables
the old block allocator.  Orlov should have better
performance - we'd like to get some feedback if it's
the contrary for you.

user_xattr Enables Extended User Attributes.  Additionally, you
need to have extended attribute support enabled in the
kernel configuration (CONFIG_EXT3_FS_XATTR).  See the
attr(5) manual page and  http://acl.bestbits.at/  to
learn more about extended attributes.

nouser_xattr Disables Extended User Attributes.

acl Enables POSIX Access Control Lists support.
Additionally, you need to have ACL support enabled in
the kernel configuration (CONFIG_EXT3_FS_POSIX_ACL).
See the acl(5) manual page and  http://acl.bestbits.at/
for more information.

noacl This option disables POSIX Access Control List
support.

reservation

noreservation

bsddf (*) Make 'df' act like BSD.
minixdf Make 'df' act like Minix.

check=none Don't do extra checking of bitmaps on mount.
nocheck

debug Extra debugging information is sent to syslog.

errors=remount-ro(*) Remount the filesystem read-only on an error.
errors=continue Keep going on a filesystem error.
errors=panic Panic and halt the machine if an error occurs.

grpid Give objects the same group ID as their creator.
bsdgroups

nogrpid (*) New objects have the group ID of their creator.
sysvgroups

resgid=n The group ID which may use the reserved blocks.

resuid=n The user ID which may use the reserved blocks.

sb=n Use alternate superblock at this location.

quota
noquota
grpquota
usrquota

bh (*) ext3 associates buffer heads to data pages to
nobh (a) cache disk block mapping information
(b) link pages into transaction to provide
    ordering guarantees.
"bh" option forces use of buffer heads.
"nobh" option tries to avoid associating buffer
heads (supported only for "writeback" mode).

Specification
=============
Ext3 shares all disk implementation with the ext2 filesystem, and adds
transactions capabilities to ext2.  Journaling is done by the Journaling
Block
Device layer.

Journaling Block Device layer
-----------------------------
The Journaling Block Device layer (JBD) isn't ext3 specific.  It was
designed
to add journaling capabilities to a block device.  The ext3 filesystem
code
will inform the JBD of modifications it is performing (called a
transaction).
The journal supports the transactions start and stop, and in case of a
crash,
the journal can replay the transactions to quickly put the partition
back into
a consistent state.

Handles represent a single atomic update to a filesystem.  JBD can
handle an
external journal on a block device.

Data Mode
---------
There are 3 different data modes:

* writeback mode
In data=writeback mode, ext3 does not journal data at all.  This mode
provides
a similar level of journaling as that of XFS, JFS, and ReiserFS in its
default
mode - metadata journaling.  A crash+recovery can cause incorrect data
to
appear in files which were written shortly before the crash.  This mode
will
typically provide the best ext3 performance.

* ordered mode
In data=ordered mode, ext3 only officially journals metadata, but it
logically
groups metadata and data blocks into a single unit called a transaction.
When
it's time to write the new metadata out to disk, the associated data
blocks
are written first.  In general, this mode performs slightly slower than
writeback but significantly faster than journal mode.

* journal mode
data=journal mode provides full data and metadata journaling.  All new
data is
written to the journal first, and then to its final location.
In the event of a crash, the journal can be replayed, bringing both data
and
metadata into a consistent state.  This mode is the slowest except when
data
needs to be read from and written to disk at the same time where it
outperforms all other modes.

Compatibility
-------------

Ext2 partitions can be easily convert to ext3, with `tune2fs -j <dev>`.
Ext3 is fully compatible with Ext2.  Ext3 partitions can easily be
mounted as
Ext2.

External Tools
==============
See manual pages to learn more.

tune2fs: create a ext3 journal on a ext2 partition with the -j flag.
mke2fs: create a ext3 partition with the -j flag.
debugfs: ext2 and ext3 file system debugger.
ext2online: online (mounted) ext2 and ext3 filesystem resizer

References
==========

kernel source: <file:fs/ext3/>
<file:fs/jbd/>

programs:  http://e2fsprogs.sourceforge.net/ http://ext2resize.sourceforge.net

useful links:  http://www.zip.com.au/~akpm/linux/ext3/ext3-usage.html http://www-106.ibm.com/developerworks/linux/library/l-fs7/ http://www-106.ibm.com/developerworks/linux/library/l-fs8/

This email and any attachment are intended solely for the addressee, are strictly confidential and may be legally privileged. If you are not the intended recipient any reading, dissemination, copying or any other use or reliance is prohibited. If you have received this email in error please notify the sender immediately by email and then permanently delete the email. Copyright reserved.

All communications, incoming and outgoing, may be recorded and are monitored for legitimate business purposes.

The security and reliability of email transmission cannot be guaranteed. It is the recipient’s responsibility to scan this e-mail and any attachment for the presence of viruses.

The Capita Group plc and its subsidiaries ("Capita") exclude all liability for any loss or damage whatsoever arising or resulting from the receipt, use or transmission of this email.

Any views or opinions expressed in this email are those of the author only.

BIG SNIP

    Your advice is way too complex. I know how to do it with my advice.

--

    Karl F. Larsen, AKA K5DI
    Linux User
    #450462    http://counter.li.org.
         Key ID = 3951B48D

Good advice. I remounted the drive with data=writeback and commit=30.
Deletions are still taking a very long time, but they are happening MUCH
faster than before.

Thanks!

Regards, K.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Karl Auer (kauer*******/~kauer/                   +61-428-957160 (mob)

GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF

Your answer did not address my problem.

My question was not about "how to delete stuff", it was about "how to
delete stuff faster". Copying a file from one drive to another,
particularly across a relatively slow link such as USB, is always going
to be slower than just deleting the file.

Steve's advice was precisely what I was looking for.

Regards, K.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Karl Auer (kauer*******/~kauer/                   +61-428-957160 (mob)

GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF

Whatever turns you on. Nothing is as fast as format however. Did you
consider that?

Karl

--

    Karl F. Larsen, AKA K5DI
    Linux User
    #450462    http://counter.li.org.
         Key ID = 3951B48D

"I don't want to delete ALL the backups - just the oldest ones - so I
can't just reformat the disk."

??

If it wasn't for that, your "advice" would be OK.

Chris

ext4 is supposed to be faster than ext3 at deleting small files, but don't
take my word on it. Look up some benchmarks on the web.
The good thing about ext4 is that you can upgrade from ext3 without
needing to reformat the drive. See http://maketecheasier.com/how-to-upgrade-from-ext3-to-ext4-w[..]

However if you do an ext3->ext4 upgrade, you don't get all of the benefits
from ext4. For example an upgraded ext3->ext4 disk doesn't use extents
like a clean formatted ext4 does. You will need to "defrag" the disk with
a tool like e4defrag, and that will take just as long (or even longer) as
copying from the old disk to a new disk.

I suggest that you do some test on a loopback filesystem first, before you
hose your backup disk. :)

--
Amedee

Your suggestion was at least workable, but not much use in my particular
case.

My requirement, clearly stated, was to delete much, but not all of the
material on the disk. For your suggestion to have been useful, I would
have had to copy off the material I wanted to keep, format the old disk,
then copy the material back.

Depending on how slow the deletions were, how fast the copy was and how
much of the old material I wanted to keep, there would be a "break-even"
point with your suggestion, where the cost in time of doing the
copy/format would have been less than the cost in time of doing the
deletions. Working out whether I was on the good side or the bad side of
that point would have been very difficult.

Thanks, K.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Karl Auer (kauer*******/~kauer/                   +61-428-957160 (mob)

GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF

So are you saying that using e4defrag will make and upgraded ext4 disk
as good as a new one?

--
Douglas E Knapp

Why do we live?

I see you've already improved on the situation.. but I wanted to suggest
a couple things you could look into.

1. make sure this really is a USB2 enclosure, and nothing is degrading
the Usb level to 1.  The quickest way I can do that is to simply start
reading from the drive and check your transfer rate.

dd if=/dev/usbdrive of=/dev/null  and use another terminal to check
iostat (though personally, I prefer using gkrellm for these kinds of
quick and dirty visual checks.)  You should be getting transfer rate of
approx 30 MB/s.

2.  If you are trying to stuff over than 10,000 small files into a
single directory, it is *vital* that the ext3 file system on your hard
drive has the dir_index option.  This should be the default on new file
systems now, but might not have been if your drive was formatted years
ago.  You will recognize this problem if file system operations (such as
deleting the backups) causes 100% system cpu utilization (one one cpu,
for smp systems.)

Check your drive with tune2fs -l /dev/usbdevice.  Make certain that
Filesystem Options line includes dir_index.  If it doesn't. use tune2fs
-O dir_index /dev/... followed with a e2fsck -D /dev/...

It would rather depend on what proportion of the files the OP wanted
to keep.  If he only wanted to keep, say, 10% of the files then Karl's
suggestion might well have been the best.  Copy off the files to be
kept, format the drive (which has the benefit of cleaning up the file
system) and copy them back.  If, however, he wanted to keep 90% then
maybe the other technique is better.  Certainly Karl's technique does
not deserve the derision that some are casting on it.

Colin

--
ubuntu-users mailing list
ubuntu-users*******
Modify settings or unsubscribe at:  https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

You are correct. Just re-read Karl's post and realized I mis-read it
slightly. Apologies to Karl, it does make some sense.

Chris

Lot of good suggestions, this may help:

http://www.unix.com/unix-dummies-questions-answers/51400-how[..]

--
Fred
www.fwrgallery.com

"Life is like linux, simple.  If you are fighting it you are doing something wrong."

If you find my advice to mount the filesystem with two additional
parameters "too complex", then really you shouldn't be attempting to
give any advice to anyone. All you suggested was cloning the drive to
another drive and then formatting the first one... what does that
achieve precisely?

The OP asked for a means of deleting files more rapidly. You got him
to clone the drive. Genius.

--
Steve
When one person suffers from a delusion it is insanity. When many
people suffer from a delusion it is called religion.

09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0

--
ubuntu-users mailing list
ubuntu-users*******
Modify settings or unsubscribe at:  https://lists.ubuntu.com/mailman/listino/ubuntu-users

Now that you ask me, I'm not entirely sure any more that the name of the
tool is e4defrag. It could be some other tool.
But yes, I am saying that essentially you can run a certain tool that will
make an upgraded disk (almost -> the actual performance difference will be
marginal for practical use) as good as a new one, AND at the same time I'm
saying that it will take more time to run this tool than just copy the
data to another disk, format, and copy the data back.
So it may not even be worth the effort.

--
Amedee

of the backups. I suggested he save the 10% he wants to keep on another
hard drive. Then reformat the original drive. Then put the 10% back on
the original drive. Delete it from the temporary hard drive.

    I stated correctly that a format will delete his files far faster than
your complex method. Let us drop this subject since he has already done
what he wanted!

73 Karl

--

    Karl F. Larsen, AKA K5DI
    Linux User
    #450462    http://counter.li.org.
         Key ID = 3951B48D