Linux Format forums Forum Index Linux Format forums
Help, discussion, magazine feedback and more
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

quickly creating / reserving space for large files, not dd?

 
Post new topic   Reply to topic    Linux Format forums Forum Index -> Programming
View previous topic :: View next topic  
Author Message
silly_xp_user



Joined: Sat Oct 22, 2005 4:55 pm
Posts: 11

PostPosted: Sat Dec 30, 2006 6:55 pm    Post subject: quickly creating / reserving space for large files, not dd? Reply with quote

Hello,

In winxp, with an NTFS disk, you can issue the command fsutil file createnew big.bin 123456789012 to create a large zeroed file and this works almost instantaneously because it does not write data to disk, it uses some win32 api calls to reserve space. This is not the same as a truly sparse file since the size on disk is the correct size.

I wondered if there was a linux equivalent. The traditional dd if=/dev/zero ... will obviously take time.

If there is no obvious way then are there other methods of quickly reserving space?

Thanks for any help given.

Hal
Back to top
View user's profile Send private message
drws
LXF regular


Joined: Sat Aug 06, 2005 11:39 am
Posts: 125

PostPosted: Sat Dec 30, 2006 7:14 pm    Post subject: RE: quickly creating / reserving space for large files, not Reply with quote

you can still use dd to create the file just do
Code:
dd if=/dev/zero of=filename bs=1M count=1 seek=50

that should create a 51Mb file by writing 1Mb of data but seeking 50meg into a file first.
ls should show the file as 51Mb, but du should show its true size (1Mb)

I set the block size to 1Mb just to make the maths easier (the seek is how many multiples of block size to skip in the output file)
You could make it bs=1 for 1 byte and then set seek accordingly.
Back to top
View user's profile Send private message
silly_xp_user



Joined: Sat Oct 22, 2005 4:55 pm
Posts: 11

PostPosted: Sat Dec 30, 2006 7:51 pm    Post subject: RE: quickly creating / reserving space for large files, not Reply with quote

That's a good start but in your example, I would want it's true size to be 51Mb and would want du to show it as 51Mb.

Hal
Back to top
View user's profile Send private message
MartyBartfast
LXF regular


Joined: Mon Aug 22, 2005 8:25 am
Posts: 804
Location: Hants, UK

PostPosted: Sat Dec 30, 2006 8:43 pm    Post subject: RE: quickly creating / reserving space for large files, not Reply with quote

Well how about a slight tweak on drws' solution:

~$ dd if=/dev/zero of=a.a bs=1k count=1000
1000+0 records in
1000+0 records out
1024000 bytes (1.0 MB) copied, 0.005958 seconds, 172 MB/s
~$ ls -l a.a
-rw-r--r-- 1 martyn martyn 1024000 2006-12-30 20:39 a.a
~$ du -b a.a
1024000 a.a
martyn@inspiron:~$
_________________
I have been touched by his noodly appendage.
Back to top
View user's profile Send private message
silly_xp_user



Joined: Sat Oct 22, 2005 4:55 pm
Posts: 11

PostPosted: Sat Dec 30, 2006 8:56 pm    Post subject: RE: quickly creating / reserving space for large files, not Reply with quote

That was plan A but as my initial post said: "The traditional dd if=/dev/zero ... will obviously take time".

The key thing is to reserve the space and not spend time writing to the file. The first example I gave reserved space for a 100GB+ file.
Back to top
View user's profile Send private message
drws
LXF regular


Joined: Sat Aug 06, 2005 11:39 am
Posts: 125

PostPosted: Sat Dec 30, 2006 10:29 pm    Post subject: RE: quickly creating / reserving space for large files, not Reply with quote

Windows seems to create a sparse looking file, in that it is created quickly without major disk writes and contains all zeroes when inspected with a hex editor.
Yet, as mentioned, its occupies whatever disk space you ask for meaning it isn't a sparse file.
I cannot create a file bigger than my available free disk space. (Which you can do using the 'dd' trick in linux. - because this is a sparse file)

At first I thought that NTFS was just updating its allocation table to say 'these blocks are in this file', but then I would expect the file to contain data (whatever was on that section of the hdd previously)

What can we call these sparse-looking zero-filled disk-consuming non-sparse files that NTFS creates? Smile

Apart from manually writing to the disk, I can't think of a way to create a real disk-occupying file quickly.
Unless you want to write 1 byte per cluster instead of the full 4k (assuming default 4k cluster size) - would this be quicker?
Anyone else got any ideas?
Back to top
View user's profile Send private message
drws
LXF regular


Joined: Sat Aug 06, 2005 11:39 am
Posts: 125

PostPosted: Sat Dec 30, 2006 10:47 pm    Post subject: RE: quickly creating / reserving space for large files, not Reply with quote

Another thought
Do you want to create a file, or just reserve disk space?
You could reserve more space for root on a partition using tune2fs. It is usually set to 5%, so non-root users see the disk full at 95% instead of 100% etc.
Back to top
View user's profile Send private message
silly_xp_user



Joined: Sat Oct 22, 2005 4:55 pm
Posts: 11

PostPosted: Sun Dec 31, 2006 11:01 am    Post subject: Re: RE: quickly creating / reserving space for large files, Reply with quote

Quote:
At first I thought that NTFS was just updating its allocation table to say 'these blocks are in this file'

Even that would be fine - how would you do it in linux?
Quote:
Unless you want to write 1 byte per cluster instead of the full 4k (assuming default 4k cluster size) - would this be quicker?

Dunno but if we scrap the requirement that it's a (completely) zereoed file then how do you do this?

Perhaps the original request query of creating a 'reserved space pseudo sparse' file might might be one for a C program? Any takers?

I may have to resort to tune2fs but would would prefer a simple file creation tool/script/program.

Hal
Back to top
View user's profile Send private message
silly_xp_user



Joined: Sat Oct 22, 2005 4:55 pm
Posts: 11

PostPosted: Sun Dec 31, 2006 11:06 am    Post subject: RE: Re: RE: quickly creating / reserving space for large fil Reply with quote

What about creating a large swap file and not using it?
Is this fast or do I still need to use dd to fill/size it?
Hal
Back to top
View user's profile Send private message
drws
LXF regular


Joined: Sat Aug 06, 2005 11:39 am
Posts: 125

PostPosted: Sun Dec 31, 2006 2:49 pm    Post subject: RE: Re: RE: quickly creating / reserving space for large fil Reply with quote

i tried creating a sparse file and then formatting for swap usage, and although it looks like a large swap file, its still only occupying a small amount of disk space. ( also tried using mkfs as well)
I've been looking a how programs that reserve disk space work and they either create the sparse file, or take ages creating a blank file.
I would have thought if there was an obvious way then some of those developers would have used it.
Back to top
View user's profile Send private message
View previous topic :: View next topic  
Display posts from previous:   
Post new topic   Reply to topic    Linux Format forums Forum Index -> Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Linux Format forums topic RSS feed 


Powered by phpBB © 2001, 2005 phpBB Group


Copyright 2011 Future Publishing, all rights reserved.


Web hosting by UKFast