Linux Format forums

Re: De-internet-archive-scripting webpages

Dutch_Master — Thu Jun 21, 2012 3:07 pm

Author: Dutch_Master

Posted: Thu Jun 21, 2012 3:07 pm

And again I find some familiar question in LFX159 this time Thx guys!
(haven't pursued it yet, got some RSI complaints by my wrist back then... )

Re: De-internet-archive-scripting webpages

Dutch_Master — Mon Mar 26, 2012 2:06 pm

Author: Dutch_Master

Posted: Mon Mar 26, 2012 2:06 pm

Thanks Nelz, give it a try later.

Re: De-internet-archive-scripting webpages

nelz — Mon Mar 26, 2012 10:34 am

Author: nelz

Posted: Mon Mar 26, 2012 10:34 am

My bad, I was working from failing memory, it is x,yd not x-yd, which means something completely different.

You can also give an extension to -i and sed will create backups of the original files with that extension. This will remove the toolbar stuff and save a backup

Code:

sed -i.bak /BEGIN\ WAYBACK\ TOOLBAR/,/END\ WAYBACK\ TOOLBAR/d Start\ page.html

You can change the links to relative with something like

Code:

sed 's/http:\/\/web.archive.org\/web\/20041103050546\/http:\/\/web.utanet.at\/smiderkr\/asr\//g'

on all files in the same directory, but it gets messy if the pages are stored in multiple subdirectories.

Re: De-internet-archive-scripting webpages

Dutch_Master — Mon Mar 26, 2012 2:32 am

Author: Dutch_Master

Posted: Mon Mar 26, 2012 2:32 am

The problem is that although the archive script puts a lot of files in the same place, it also hard-links all links in a page, using absolute links (with the http header). However, when storing the files I also introduced some issues, only to become apparent when I opened the html code.... Right now, I think your script will remove the bulk of the added code and I'd have to manually edit the hard-links back to relative links. Thanks again Nelz!

[edit: cried victory too soon, after replacing the x and y with numerical values I got the error

Code:

sed: -e expression #1, char 3: unknown command: `-'

I've got as far as

Code:

for i in *.html; do sed i\ {14-216}d $i; done

This clears the files completely. Luckily I got a backup.... ]

[edit2: here's a simple sample]

Re: De-internet-archive-scripting webpages

nelz — Mon Mar 26, 2012 1:55 am

Author: nelz

Posted: Mon Mar 26, 2012 1:55 am

If the code is in the same place in each file, you can remove a range of line with

Code:

for i in *.html; do
sed -i x-yd $i
done

Where x and y are the first and last lines of the script. Otherwise we'd need to see an example to see how to identify the lines to be deleted.

De-internet-archive-scripting webpages

Dutch_Master — Sun Mar 25, 2012 10:18 pm

Author: Dutch_Master

Posted: Sun Mar 25, 2012 10:18 pm

Some time ago I downloaded a website from the internet archive pages as it turned out the owner discontinued it. They insert some ***** scripting I distaste greatly and it keeps on linking to the archive. I could remove all instances by hand in the html code, but with 100+ pages, I'd think there's a better (quicker!) solution. I assume sed or awk are required, but knowing nought about either, what's the best oneliner (or script, that's fine) that get me running? (s'cuse the pun )