Hello. Me and a few friends are attempting to backup every files from AndroidFileHost, and we need some help in doing so.
For those who haven’t heard of it, AndroidFileHost is a website that hosts various Android related files. It’s one of the last surviving large Android related file serving sites, and holds a LOT of rare files especially for older android devices. (rip d-h.st) Despite being such a valuable site, it hasn’t been well maintained for the past few years. Their Xitter account’s last update is from around 2022, and the owner isn’t replying to any e-mails. The site has been extremely unstable with various issues, most recently no file could ever be downloaded from it for about a month. Luckily, it has been (kind of) solved for now, and most (not all, about 20% files are still gone) files are back online now. However, it’s clear this site needs a backup.
I have scraped their website which gives us the unique ID and MD5 hash for every files available on the site. Now, using this ID we can automate the process of requesting mirror links, downloading them and checking for integrity. (Please check an example file to understand how their system works – https://androidfilehost.com/?fid=745425885120701975 )
The sum of every file sizes we know is roughly 180TB. It’s impossible to download this on a single machine, so I’ve developed a “tracker” system to concurrently download multiple files using different machines. The tracker server keeps a list of every known file IDs (btw, it’s 256,640 files which is a bit less than 277,467 displayed on their main page. I believe it includes deleted files as well but not sure atm), assign it to each clients that request and appropriately mark the file as downloaded. The system is pretty robust now, so our plan is working great. Except that our internet is pretty slow and we can’t afford 180TB instantly.
By talking to friends and their friends, we’ve got quite a few people willing to help a bit here. Unfortunately many of them lack storage space, so they need to keep downloading from AFH and uploading to my server. This works for a few clients, but not for many. The “my server” here every client uploads to have 500Mbps internet, and it gets terribly slow pretty quick. Plus, 180TB of storage isn’t really cheap and easy to afford.
Ideally, we need to get people with faster internet speeds (I’m in asia, so not the best place to fetch files from AFH servers mostly around Europe and America) and more storage space. If you have some bandwidth or storage to share, it would greatly help us.
I’m sorry if a post like this isn’t welcomed here, if so please feel free to remove it. Thanks for reading this post.
I haven’t tried. As my script isn’t downloading to WARCs and not compatible with their tracker I believe someone would have to create a new script to run with their infrastructure… Though, maybe even if it can’t be an AT project someone might help, so I guess I’ll try asking.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: [email protected]
5. Engage respectfully: Harassment, flamebaiting, bad faith engagement, or agenda posting will result in your posts being removed. Excessive violations will result in temporary or permanent ban, depending on severity.
6. Memes are not allowed to be posts, but are allowed in the comments.
7. Posts from clickbait sources are heavily discouraged. Please de-clickbait titles if it needs to be submitted.
8. Submission statements of any length composed of your own thoughts inside the post text field are mandatory for any microblog posts, and are optional but recommended for article/image/video posts.
Maybe also cross-post in data-hoarding and archival communities? Seen some in the time I’ve been here on the fediverse.
Great idea
Have you worked with https://wiki.archiveteam.org/index.php/Main_Page?
I haven’t tried. As my script isn’t downloading to WARCs and not compatible with their tracker I believe someone would have to create a new script to run with their infrastructure… Though, maybe even if it can’t be an AT project someone might help, so I guess I’ll try asking.
If it becomes compatible, that’s a lot of bandwidth and storage available.