Linux Format forums

by **Bazza** » Sat Jan 12, 2013 9:56 am

This is for shell programming experts...

The code below works fine but...

It is UGLY, (just like me ;o)...

I need generate random length binary files on the fly, so to test I used a simple for loop to generate all of the _bytes_ from 0 to 255.

As you can see from the simple code I am doing it in three steps.

I have searched the WWW on how to generate random length binary files on the fly but nothing came useful at all...

So I am asking for help, here is the code and a copy of the results from the shell as comments in the code...

Can it be simplified? If so, point me in the right direction...

TIA, Bazza...

Code: Select all: # !/bin/sh for char in {0..255} do printf '\\\\x'"%02x" $char >> BinaryFile.txt done read somefile < BinaryFile.txt printf "$somefile" > BinaryFile.dat ls -al BinaryFile.* hexdump -C BinaryFile.txt hexdump -C BinaryFile.dat # Last login: Sat Jan 12 09:03:49 on ttys000 # Barrys-MacBook-Pro:~ barrywalker$ /Users/barrywalker/binary.sh # -rw-r--r-- 1 barrywalker staff 256 12 Jan 09:15 BinaryFile.dat # -rw-r--r-- 1 barrywalker staff 1280 12 Jan 09:15 BinaryFile.txt # 00000000 5c 5c 78 30 30 5c 5c 78 30 31 5c 5c 78 30 32 5c |\\x00\\x01\\x02\| # 00000010 5c 78 30 33 5c 5c 78 30 34 5c 5c 78 30 35 5c 5c |\x03\\x04\\x05\\| # 00000020 78 30 36 5c 5c 78 30 37 5c 5c 78 30 38 5c 5c 78 |x06\\x07\\x08\\x| # 00000030 30 39 5c 5c 78 30 61 5c 5c 78 30 62 5c 5c 78 30 |09\\x0a\\x0b\\x0| # 00000040 63 5c 5c 78 30 64 5c 5c 78 30 65 5c 5c 78 30 66 |c\\x0d\\x0e\\x0f| # 00000050 5c 5c 78 31 30 5c 5c 78 31 31 5c 5c 78 31 32 5c |\\x10\\x11\\x12\| # 00000060 5c 78 31 33 5c 5c 78 31 34 5c 5c 78 31 35 5c 5c |\x13\\x14\\x15\\| # 00000070 78 31 36 5c 5c 78 31 37 5c 5c 78 31 38 5c 5c 78 |x16\\x17\\x18\\x| # 00000080 31 39 5c 5c 78 31 61 5c 5c 78 31 62 5c 5c 78 31 |19\\x1a\\x1b\\x1| # 00000090 63 5c 5c 78 31 64 5c 5c 78 31 65 5c 5c 78 31 66 |c\\x1d\\x1e\\x1f| # 000000a0 5c 5c 78 32 30 5c 5c 78 32 31 5c 5c 78 32 32 5c |\\x20\\x21\\x22\| # 000000b0 5c 78 32 33 5c 5c 78 32 34 5c 5c 78 32 35 5c 5c |\x23\\x24\\x25\\| # 000000c0 78 32 36 5c 5c 78 32 37 5c 5c 78 32 38 5c 5c 78 |x26\\x27\\x28\\x| # 000000d0 32 39 5c 5c 78 32 61 5c 5c 78 32 62 5c 5c 78 32 |29\\x2a\\x2b\\x2| # 000000e0 63 5c 5c 78 32 64 5c 5c 78 32 65 5c 5c 78 32 66 |c\\x2d\\x2e\\x2f| # 000000f0 5c 5c 78 33 30 5c 5c 78 33 31 5c 5c 78 33 32 5c |\\x30\\x31\\x32\| # 00000100 5c 78 33 33 5c 5c 78 33 34 5c 5c 78 33 35 5c 5c |\x33\\x34\\x35\\| # 00000110 78 33 36 5c 5c 78 33 37 5c 5c 78 33 38 5c 5c 78 |x36\\x37\\x38\\x| # 00000120 33 39 5c 5c 78 33 61 5c 5c 78 33 62 5c 5c 78 33 |39\\x3a\\x3b\\x3| # 00000130 63 5c 5c 78 33 64 5c 5c 78 33 65 5c 5c 78 33 66 |c\\x3d\\x3e\\x3f| # 00000140 5c 5c 78 34 30 5c 5c 78 34 31 5c 5c 78 34 32 5c |\\x40\\x41\\x42\| # 00000150 5c 78 34 33 5c 5c 78 34 34 5c 5c 78 34 35 5c 5c |\x43\\x44\\x45\\| # 00000160 78 34 36 5c 5c 78 34 37 5c 5c 78 34 38 5c 5c 78 |x46\\x47\\x48\\x| # 00000170 34 39 5c 5c 78 34 61 5c 5c 78 34 62 5c 5c 78 34 |49\\x4a\\x4b\\x4| # 00000180 63 5c 5c 78 34 64 5c 5c 78 34 65 5c 5c 78 34 66 |c\\x4d\\x4e\\x4f| # 00000190 5c 5c 78 35 30 5c 5c 78 35 31 5c 5c 78 35 32 5c |\\x50\\x51\\x52\| # 000001a0 5c 78 35 33 5c 5c 78 35 34 5c 5c 78 35 35 5c 5c |\x53\\x54\\x55\\| # 000001b0 78 35 36 5c 5c 78 35 37 5c 5c 78 35 38 5c 5c 78 |x56\\x57\\x58\\x| # 000001c0 35 39 5c 5c 78 35 61 5c 5c 78 35 62 5c 5c 78 35 |59\\x5a\\x5b\\x5| # 000001d0 63 5c 5c 78 35 64 5c 5c 78 35 65 5c 5c 78 35 66 |c\\x5d\\x5e\\x5f| # 000001e0 5c 5c 78 36 30 5c 5c 78 36 31 5c 5c 78 36 32 5c |\\x60\\x61\\x62\| # 000001f0 5c 78 36 33 5c 5c 78 36 34 5c 5c 78 36 35 5c 5c |\x63\\x64\\x65\\| # 00000200 78 36 36 5c 5c 78 36 37 5c 5c 78 36 38 5c 5c 78 |x66\\x67\\x68\\x| # 00000210 36 39 5c 5c 78 36 61 5c 5c 78 36 62 5c 5c 78 36 |69\\x6a\\x6b\\x6| # 00000220 63 5c 5c 78 36 64 5c 5c 78 36 65 5c 5c 78 36 66 |c\\x6d\\x6e\\x6f| # 00000230 5c 5c 78 37 30 5c 5c 78 37 31 5c 5c 78 37 32 5c |\\x70\\x71\\x72\| # 00000240 5c 78 37 33 5c 5c 78 37 34 5c 5c 78 37 35 5c 5c |\x73\\x74\\x75\\| # 00000250 78 37 36 5c 5c 78 37 37 5c 5c 78 37 38 5c 5c 78 |x76\\x77\\x78\\x| # 00000260 37 39 5c 5c 78 37 61 5c 5c 78 37 62 5c 5c 78 37 |79\\x7a\\x7b\\x7| # 00000270 63 5c 5c 78 37 64 5c 5c 78 37 65 5c 5c 78 37 66 |c\\x7d\\x7e\\x7f| # 00000280 5c 5c 78 38 30 5c 5c 78 38 31 5c 5c 78 38 32 5c |\\x80\\x81\\x82\| # 00000290 5c 78 38 33 5c 5c 78 38 34 5c 5c 78 38 35 5c 5c |\x83\\x84\\x85\\| # 000002a0 78 38 36 5c 5c 78 38 37 5c 5c 78 38 38 5c 5c 78 |x86\\x87\\x88\\x| # 000002b0 38 39 5c 5c 78 38 61 5c 5c 78 38 62 5c 5c 78 38 |89\\x8a\\x8b\\x8| # 000002c0 63 5c 5c 78 38 64 5c 5c 78 38 65 5c 5c 78 38 66 |c\\x8d\\x8e\\x8f| # 000002d0 5c 5c 78 39 30 5c 5c 78 39 31 5c 5c 78 39 32 5c |\\x90\\x91\\x92\| # 000002e0 5c 78 39 33 5c 5c 78 39 34 5c 5c 78 39 35 5c 5c |\x93\\x94\\x95\\| # 000002f0 78 39 36 5c 5c 78 39 37 5c 5c 78 39 38 5c 5c 78 |x96\\x97\\x98\\x| # 00000300 39 39 5c 5c 78 39 61 5c 5c 78 39 62 5c 5c 78 39 |99\\x9a\\x9b\\x9| # 00000310 63 5c 5c 78 39 64 5c 5c 78 39 65 5c 5c 78 39 66 |c\\x9d\\x9e\\x9f| # 00000320 5c 5c 78 61 30 5c 5c 78 61 31 5c 5c 78 61 32 5c |\\xa0\\xa1\\xa2\| # 00000330 5c 78 61 33 5c 5c 78 61 34 5c 5c 78 61 35 5c 5c |\xa3\\xa4\\xa5\\| # 00000340 78 61 36 5c 5c 78 61 37 5c 5c 78 61 38 5c 5c 78 |xa6\\xa7\\xa8\\x| # 00000350 61 39 5c 5c 78 61 61 5c 5c 78 61 62 5c 5c 78 61 |a9\\xaa\\xab\\xa| # 00000360 63 5c 5c 78 61 64 5c 5c 78 61 65 5c 5c 78 61 66 |c\\xad\\xae\\xaf| # 00000370 5c 5c 78 62 30 5c 5c 78 62 31 5c 5c 78 62 32 5c |\\xb0\\xb1\\xb2\| # 00000380 5c 78 62 33 5c 5c 78 62 34 5c 5c 78 62 35 5c 5c |\xb3\\xb4\\xb5\\| # 00000390 78 62 36 5c 5c 78 62 37 5c 5c 78 62 38 5c 5c 78 |xb6\\xb7\\xb8\\x| # 000003a0 62 39 5c 5c 78 62 61 5c 5c 78 62 62 5c 5c 78 62 |b9\\xba\\xbb\\xb| # 000003b0 63 5c 5c 78 62 64 5c 5c 78 62 65 5c 5c 78 62 66 |c\\xbd\\xbe\\xbf| # 000003c0 5c 5c 78 63 30 5c 5c 78 63 31 5c 5c 78 63 32 5c |\\xc0\\xc1\\xc2\| # 000003d0 5c 78 63 33 5c 5c 78 63 34 5c 5c 78 63 35 5c 5c |\xc3\\xc4\\xc5\\| # 000003e0 78 63 36 5c 5c 78 63 37 5c 5c 78 63 38 5c 5c 78 |xc6\\xc7\\xc8\\x| # 000003f0 63 39 5c 5c 78 63 61 5c 5c 78 63 62 5c 5c 78 63 |c9\\xca\\xcb\\xc| # 00000400 63 5c 5c 78 63 64 5c 5c 78 63 65 5c 5c 78 63 66 |c\\xcd\\xce\\xcf| # 00000410 5c 5c 78 64 30 5c 5c 78 64 31 5c 5c 78 64 32 5c |\\xd0\\xd1\\xd2\| # 00000420 5c 78 64 33 5c 5c 78 64 34 5c 5c 78 64 35 5c 5c |\xd3\\xd4\\xd5\\| # 00000430 78 64 36 5c 5c 78 64 37 5c 5c 78 64 38 5c 5c 78 |xd6\\xd7\\xd8\\x| # 00000440 64 39 5c 5c 78 64 61 5c 5c 78 64 62 5c 5c 78 64 |d9\\xda\\xdb\\xd| # 00000450 63 5c 5c 78 64 64 5c 5c 78 64 65 5c 5c 78 64 66 |c\\xdd\\xde\\xdf| # 00000460 5c 5c 78 65 30 5c 5c 78 65 31 5c 5c 78 65 32 5c |\\xe0\\xe1\\xe2\| # 00000470 5c 78 65 33 5c 5c 78 65 34 5c 5c 78 65 35 5c 5c |\xe3\\xe4\\xe5\\| # 00000480 78 65 36 5c 5c 78 65 37 5c 5c 78 65 38 5c 5c 78 |xe6\\xe7\\xe8\\x| # 00000490 65 39 5c 5c 78 65 61 5c 5c 78 65 62 5c 5c 78 65 |e9\\xea\\xeb\\xe| # 000004a0 63 5c 5c 78 65 64 5c 5c 78 65 65 5c 5c 78 65 66 |c\\xed\\xee\\xef| # 000004b0 5c 5c 78 66 30 5c 5c 78 66 31 5c 5c 78 66 32 5c |\\xf0\\xf1\\xf2\| # 000004c0 5c 78 66 33 5c 5c 78 66 34 5c 5c 78 66 35 5c 5c |\xf3\\xf4\\xf5\\| # 000004d0 78 66 36 5c 5c 78 66 37 5c 5c 78 66 38 5c 5c 78 |xf6\\xf7\\xf8\\x| # 000004e0 66 39 5c 5c 78 66 61 5c 5c 78 66 62 5c 5c 78 66 |f9\\xfa\\xfb\\xf| # 000004f0 63 5c 5c 78 66 64 5c 5c 78 66 65 5c 5c 78 66 66 |c\\xfd\\xfe\\xff| # 00000500 # 00000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................| # 00000010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................| # 00000020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f | !"#$%&'()*+,-./| # 00000030 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f |0123456789:;<=>?| # 00000040 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f |@ABCDEFGHIJKLMNO| # 00000050 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f |PQRSTUVWXYZ[\]^_| # 00000060 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f |`abcdefghijklmno| # 00000070 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f |pqrstuvwxyz{|}~.| # 00000080 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f |................| # 00000090 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f |................| # 000000a0 a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af |................| # 000000b0 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf |................| # 000000c0 c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf |................| # 000000d0 d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df |................| # 000000e0 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef |................| # 000000f0 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff |................| # 00000100 # Barrys-MacBook-Pro:~ barrywalker$ _

by **xenopeek** » Sat Jan 12, 2013 10:50 am

You can read from /dev/urandom to get random bytes. For example, to create a file called "yourfile" in the current directory with 256 random bytes do:

Code: Select all: head -c 256 </dev/urandom >yourfile

by **Bazza** » Sat Jan 12, 2013 11:48 am

Hi xenopeek...

I already know that as I use it for generating white noise sometimes using /dev/dsp but...

I need a random LENGTH of a binary file not a file of randomly generated bytes of any length...

I WILL know what is in the binary file but I will not know its length at run time.

For example the binary file may be 8192 bytes in length...

I may want to extract say 64 bytes from any section of it or 128 bytes from equispaced jumps inside this binary file... This is not my problem it is generating this known binary file from a text mode format... I want to be able to save my KNOWN binary file straight into binary and NOT have the text mode version of it generated too...

Does this make more sense...

by **MartyBartfast** » Sat Jan 12, 2013 1:08 pm

Is this what you're looking for?

Code: Select all: martyn@inspiron:~$ cat test.sh #!/bin/sh # zero the binary file >Binaryfile.dat # get random size, up to 100 chars numchars=$[( $RANDOM % 100 )] echo "Producing a file of $numchars chars" for ((r=1;r<${numchars};r++)) do char=`printf '\\\\x'"%02x" $r` printf "$char" >> Binaryfile.dat done martyn@inspiron:~$ sh test.sh Producing a file of 40 chars martyn@inspiron:~$ hexdump -C Binaryfile.dat 00000000 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 |................| 00000010 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 |............... | 00000020 21 22 23 24 25 26 27 |!"#$%&'| 00000027

by **Bazza** » Sat Jan 12, 2013 1:43 pm

Hi MBF...

Cool, the backticks, I forgot entirely about those and as soon as I saw them I immediately thought of AMIGA CLI...

This is more or less exactly what I want...

You are a star...

I have not tested it yet but from what I see it will do what I want...

Many thanks chap...

EDIT:-

Hi MBF

Those backticks did the trick. I am now scanning through any binary file and creating a new one in the modes described above selecting offset, jump and size...

Great stuff, many thanks matey...

by **nelz** » Sat Jan 12, 2013 5:22 pm

Use $(command) instead of `command` - you'll find it much easier to understand when you look at your script in six months.

Also, the $(...) method can be nested which can be done with backticks but only with the application of seriously ugly amounts of leaning toothpicks. If you want your code to look that bad, you may as well start using Perl

by **Bazza** » Sat Jan 12, 2013 5:41 pm

Hi nelz...

> "...seriously ugly amounts of leaning toothpicks."

LOL!

I haven't done shell scripting for a long while and it amazes me how much I have forgotten...

I shall be calling on you guys again soon no doubt...

As for the backticks it all came flooding back from the AMIGA OS days and within an hour or so I am manipulating binary files for a project that will be released on here in the not too distant future...

;o)

by **xenopeek** » Sun Jan 13, 2013 9:07 am

So combining the $RANDOM bit, you can collapse the script to one command that does the same

Code: Select all: head -c $[( $RANDOM % 100 )] </dev/urandom >Binaryfile.dat

That line generates a file called Binaryfile.dat of random length, between 0 and 99 characters, and fills it with random bytes.

by **Bazza** » Sun Jan 13, 2013 3:51 pm

Hi MBF...

Well here it is the DEMO using your backticks technique...

A known binary file and my wanted binary file.

Believe it or not my wanted file WILL become the _timebase_ of a shell AudioScope that I have already started...

I mentioned you in the comments... ;o)

Thanks a lot...

Watch out for wordwrapping, etc...

Code: Select all: # !/bin/sh # # BFM.sh # A very simple DEMO Binary File Manipulation shell code issued entirely as Public Domain. # # Apologies for any typos... # # This is a DEMO to show how to generate binaryfiles in the shell. # I need this facility for a kids project I am doing. It will be posted onto here in # the not too distant future... # # To run, ensure the script is executable and from a terminal type:- # # *****$ /absolute/path/to/BFM.sh<CR> # # And away you go... # # Many thanks to MartyBartFast of the Linux Format forums for reminding me of the backticks... ;o) # # Written so that kids and newbies can understand what is going on... # # $VER: BFM.sh_Version_0.00.10_Public_Domain_2013_B.walker_G0LCU. # # Enjoy finding simple solutions to often very difficult problems... # These two files WILL be generated inside YOUR default /directory/drawer/folder/... # Zero them as a starter JUST for this DEMO... > SomeBinaryFile.dat > MyBinaryFile.dat # This loop is to give a usable 256 byte binary file only from 0x00 to 0xFF continuous... clear for character in {0..255} do # Note:- The backticks and the four escape characters are required... char=`printf '\\\\x'"%02x" $character` printf "$char" >> SomeBinaryfile.dat done # Do a hexadecimal dump to prove the file is binary and 256 bytes in size... printf "\nOffset. Hexadecimal Dump. ASCII Dump.\n" hexdump -C SomeBinaryFile.dat # Now this EXAMPLE loop is the working loop. The three variables set the "start", (_offset_), of where you want # your file to start, the "jump" between each _sample_ and the last_byte_limit to where you want to finish... # IMPORTANT!!! There is NO error detection or correction in the code as it is a DEMO so don't come back and # say that you can crash it, e.g. setting the "last_byte_limit" to -1 for example... # Also although I am using a "for" loop for this DEMO you could select a single byte manually and code accordingly... # # The _variables_ for this DEMO. The jump is set to every seventh byte for this DEMO. start_offset=4 jump=7 last_byte_limit=200 # _hexdump_ is used to select the byte(s) according to this DEMO "for" loop... for subscript in $( seq $start_offset $jump $last_byte_limit ) do number=`hexdump -n1 -s$subscript -v -e '1/1 "%u"' SomeBinaryFile.dat` char=`printf '\\\\x'"%02x" $number` printf "$char" >> MyBinaryFile.dat done # Now display it... printf "\nOffset. Hexadecimal Dump. ASCII Dump.\n" hexdump -C MyBinaryFile.dat # A terminal window dump of what should occur... # # Offset. Hexadecimal Dump. ASCII Dump. # 00000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................| # 00000010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................| # 00000020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f | !"#$%&'()*+,-./| # 00000030 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f |0123456789:;<=>?| # 00000040 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f |@ABCDEFGHIJKLMNO| # 00000050 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f |PQRSTUVWXYZ[\]^_| # 00000060 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f |`abcdefghijklmno| # 00000070 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f |pqrstuvwxyz{|}~.| # 00000080 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f |................| # 00000090 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f |................| # 000000a0 a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af |................| # 000000b0 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf |................| # 000000c0 c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf |................| # 000000d0 d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df |................| # 000000e0 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef |................| # 000000f0 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff |................| # 00000100 # # Offset. Hexadecimal Dump. ASCII Dump. # 00000000 04 0b 12 19 20 27 2e 35 3c 43 4a 51 58 5f 66 6d |.... '.5<CJQX_fm| # 00000010 74 7b 82 89 90 97 9e a5 ac b3 ba c1 c8 |t{...........| # 0000001d # Barrys-MacBook-Pro:~ barrywalker$ # # BFM.sh DEMO end... # Enjoy finding simple solutions to often very difficult problems...

Home About Buy online Blog Archives Newsletters Subscribe Contact

Linux Format forums

I am stuck...

I am stuck...

Who is online