Comments (52)

• cellofellow

  Do you plan to run it as a public tracker or as a private tracker? If private, you could integrate it into the Mises.org Community framework.

  The real experts in torrents are in my opinion The Pirate Bay, though Demonoid isn't bad either. Maybe you could use some of their expertise.

  The question is: what is a "bittorrent server"? Is it a tracker, a seed, a torrent file directory? You could roll all three into the same box but it wouldn't be advisable in my opinion. A tracker needs to have stellar uptime, a seed can go on and off as needed, and a directory is just a fancy website. While I'm no BitTorrent expert I'll venture a guess that tracker software for Windows Server is going to be a tough call. Most (if not all) torrent trackers run on Linux.

  Just a few thoughts, this is a great idea and will gladly seed some torrents for you. :)

  -Josh

  Published: February 20, 2009 2:57 PM

• Just saying

  Jeffrey,

  It would be very helpful if you could compile a list of the Austrian and libertarian works whose copyrights you have been unable to obtain for official reprinting by LvMI.

  You could publish this list along with a polite note urging your readers not to violate their copyright by scanning them and making them available as bittorrents. I'm sure that the last thing that any anarcho-capitalist wants is to share materials which, however useful, would be distributed in violation of strict laws against such activity. Please help them to avoid the embarrassment of inadvertently doing so.

  Published: February 20, 2009 3:12 PM

• MTB

  I'd be more than willing to help with the seeding. My up speed is pretty good, and my PC stays on 24/7.

  Published: February 20, 2009 3:13 PM

• Mittybrah

  http://bnbteasytracker.sourceforge.net/

  BNBT EasyTracker is a packaged BitTorrent Tracker Installer for Windows, the comes preconfigured with what is considered to be a default configuration and includes an easy-to-use frontend for modifying the tracker's configuration file. The concept for a pre-packaged tracker was originally conceived by Hyrle, the Master of Ceremonies at filesoup.com.

  Published: February 20, 2009 3:14 PM

• Mittybrah

  It appears BNBT EasyTracker is a very dead project. http://en.wikipedia.org/wiki/EasyTracker

  Published: February 20, 2009 3:22 PM

• AJ Witoslawski

  I'll seed when you get the torrents up, because I personally do not know how to upload them on torrent sites... lol

  Published: February 20, 2009 3:22 PM

• Marco

  I'd certainly use The Pirate Bay as a tracker. Their uptime is unbelievable especially considering how often they get raided by the cops.

  I guess all that would be needed then would be to set up a table on your DBMS with the name of the work, URL of the .torrent file, short description and a responsive search/filter tool.

  Anyway, I applaud the initiative.

  Published: February 20, 2009 3:24 PM

• J.K.

  I have an institutional connection here (~40Mbps), and would be happy to help seed.

  Published: February 20, 2009 3:34 PM

• John Delano

  I can seed, but I don't want to do all 80GB at once. Is this going to be broken up into chunk of varying types of information? Or is it all 80GB in one torrent?

  
  I suppose I could just restrict bandwidth if I have it all 80GB.

  Published: February 20, 2009 3:43 PM

• Egosumabbas

  I'd be happy to upload and seed torrents on Demonoid in the meantime (with your permission of course).

  There are also some trackers which are purposely designed to carry public domain / creative commons / copyleft / etc type content, so all you would have to do is just link to that site's torrent and have a machine dedicated for seeding content.

  Also, I wouldn't worry too much about running out of bandwidth when torrents are initially posted, as content is shared as soon as a few users connect, though some volunteer seeders would definitely help.

  Published: February 20, 2009 3:47 PM

• David Veksler

  > "Is this going to be broken up into chunk of varying types of information? Or is it all 80GB in one torrent?"

  That's why I need some expert help to figure out how to do such things.

  Published: February 20, 2009 3:49 PM

• Egosumabbas

  "> "Is this going to be broken up into chunk of varying types of information? Or is it all 80GB in one torrent?"

  That's why I need some expert help to figure out how to do such things."

  Ideally you don't want to split up the whole collection of mises content into torrents of 10 megabytes or lower.

  It would be helpful to split up the mises site into several logical parts as 80 gigabytes is huge.

  For example, you could split the content by author, or by media type.

  You know what... a librarian might actually be more useful. I'll ask my wife what she thinks since she has an MS in library science :)

  Published: February 20, 2009 4:05 PM

• Egosumabbas

  Come to think of it... it doesn't matter how you split it up, as torrent files are just metadata--indices if you will.

  Ideally you would have one machine with all the data on it--it will be the seeder. You could have any number of torrents that cross reference each other, as the torrent will simply point to a file or a set of files on the seed machine.

  So you could have a mega-torrent with everything in it, or a series of torrents with sub-divided content.

  A tracker or host directory wouldn't be necessary per se, as you could use another site to host the torrents and track them (as I mentioned above).

  Published: February 20, 2009 4:10 PM

• David C

  Just out of curiosity, is that 80 GB a single database dump? is it a bunch of files from a large filesystem? Is it compressed? and how? do you have a dvd dump of it?

  I know of a linux command called split, that will split a large file into several parts, don't know of a windows equiv though.

  You know, at 80GB, you might be better off sending out dvd's - at first, maybe at a nominal cost.
  

  Published: February 20, 2009 4:30 PM

• David C

  One more thing, I assume you have a plan to dump the code and db in a way that's usable, without giving out sensitive user and password info? Will there be instructions?

  Published: February 20, 2009 4:37 PM

• David Veksler

  > Just out of curiosity, is that 80 GB a single database dump?

  The database is 180MB. We have 2-3GB of PDFs, a few MB of html, and the rest is MP3, AAC and MPG files.

  > I assume you have a plan to dump the code and db in a way that's usable, without giving out sensitive user and password info?

  Yes, unless someone volunteers, I will share the database once I write the scripts to do so.

  Published: February 20, 2009 4:49 PM

• Egosumabbas

  One thing to note (I got your email by the way, I'll reply this evening):

  Bittorrent is rather poor at serving dynamic content. So it's good for audio, video, and PDF, but not so great for databases and websites. Any dynamic content would be "snapshots" released on a regular basis.

  Published: February 20, 2009 5:10 PM

• Robert C

  I don't know much about setting up a tracker, but I do download and seed Linux distributions via BitTorrent. I'd be happy to add some LvMI publications to my seedings.

  You could very easily make one torrent for each filetype... one for PDF's, one for videos, etc. You could even offer a single torrent with everything in it, since some BT clients like uTorrent allow users to select which files within a torrent to actually download.

  Published: February 20, 2009 5:39 PM

• Mittybrah

  Opentracker is the indexing software (Tracker) powering The Pirate Bay. http://erdgeist.org/arts/software/opentracker/
  

  Published: February 20, 2009 5:49 PM

• David Bratton

  You don't need a tracker. Just use Pirate Bay (assuming they're not all in jail by the end of next week - they are in court now). Make sure the tracker you choose is a public tracker - one that doesn't require a password. Demonoid for example is private and wouldn't work.

  All you need is a client such as uTorrent or BitComet. Post your torrents to PirateBay, then copy the link onto your own site.

  You already have the media files organized into logical sections on the site. That would probably be a good way to organize them into individual torrents too. If they are already organized into directories then you just need to create one torrent for each directory.

  Don't zip, rar, or otherwise compress or munge up the contents of a multi-file torrent. Some downloaders will want to select individual files and some clients allow you to cherry pick within a torrent.

  FYI some BitTorrent lingo:
  An open tracker is one that allows anyone with access to it to post torrents. A closed tracker only allows admins to post torrents. A private tracker requires a password for access and a public tracker doesn't.
  Pirate bay is open and public.
  Demonoid is open and private.
  EZTV is closed and public.
  Xtremespeeds I believe is closed and private.
  

  Published: February 20, 2009 6:30 PM

• alansmithee

  has noone heard of the Pirate Bay trial going on at the moment? they are up shit creek because they 'aided' the distribution of copyrighted material..

  Published: February 20, 2009 7:42 PM

• James R

  Using The Pirate Bay as a tracker could have both positive and negative repercussions.

  The Pirate Bay tends to be viewed as subversive, because they host many trackers for copyrighted or otherwise IP-protected material. Using The Pirate Bay to host trackers for LvMI materials would allow the enemies of Austrian economics to cast aspersions on LvMI via association.

  Then again, in a very real sense, Austrian economics is subversive, in that it is attacking the State—not with guns, but with ideas. And to the State, that makes Austrian economics even more dangerous: for the State can defend itself against guns, but not against ideas.

  In the back of my mind, though, is the same question I've been wondering since the announcement yesterday that LvMI is going open source: what are you really attempting to accomplish?

  In the age of information technology, it is a common but nonetheless grave error to start sifting through various pieces of technology before defining the problem one is attempting to solve.

  If the problem you are attempting to solve is simply that members of the LvMI staff don't have enough time to add new material to mises.org as fast as they'd like to do so, then your solution is to find a model that can harness the power of volunteers. That solution probably will not include BitTorrent, because that's not really the problem BitTorent was designed to solve.

  If the problem you are attempting to solve is that you fear the State will attempt to "shut down" the LvMI and/or the mises.org web site, then you have an entirely different problem, and the potential solutions will also be different.

  I implore you: instead of randomly throwing pieces of mises.org "over the fence" and hoping that someone comes along and does something useful with them, or instead of asking advice on how to configure [X] or [Y] or [Z] technology, take a step back, forget about the technology for a moment, clear your head (get out of the office and take a walk, even), and then answer this question in your mind: in the most general terms, what is the problem you are trying to solve? What is your goal?

  Then come back to the office, sit down, and explain to the rest of us what your goal is, not the technology that you think will help realize it. Put that explanation in a form in which others can both link to it and comment on it (e.g., the wiki). End the explanation with a polite and sincere request for assistance.

  The collective power of the Internet is an amazing thing. (Ludwig von Mises would surely weep tears of joy were he to witness it.) But you have to help us to help you.

  Published: February 20, 2009 7:59 PM

• Conza88

  I have nearly all the audio files from the media section.

  Would I have to download them again?

  A few like the Cost of War and others, when you download them from the site - they are blank, with no info.

  I had too check them all and add in content from the site. All is good. Just whoever hosts a torrent, should make it perfect. :)

  There would be a few good ways to package the files. Ultimately, them all in folders that mirror the site, just as I've done is pretty helpful for when there are additions to sections, you simply add to it.

  Collections by author, category, section, type could all be made. No worries.

  Published: February 20, 2009 11:06 PM

• Peter

  Bittorrent is rather poor at serving dynamic content. So it's good for audio, video, and PDF, but not so great for databases and websites.

  That and the question about whether it's one big 80Gb lump....I'd assume they're only talking about sharing the individual files (PDFs, videos, etc.), like the sensible people they are, not the "entire site". Thousands of individual torrents, not one huge torrent. At least I hope so, because the latter would be quite insane!

  Published: February 20, 2009 11:29 PM

• Deefburger

  I'm grabbing the content for the VM linux project any way. I'll see about creating a torrent for it as well since that is how I want to initialize and distribute the VMs.

  Published: February 20, 2009 11:37 PM

• Julien Couvreur

  For audio and video files, why not upload them to Youtube, Vimeo or some other free (advertising funded) service?

  Published: February 20, 2009 11:38 PM

• Curt Howland

  The information is already indexed. "The Rothbard Collection", "The Mises Collection", Mises University 2008, Mises University 2007, Reassessing The Presidency, etc.

  So a single torrent would be, "Mises University 2008.torrent"

  Or, "Daily Articles in HTML 2008.torrent"

  Sadly, I don't have 80GB available for mirroring.

  Published: February 21, 2009 8:00 AM

• MTB

  Here's how I'd organize the media.

  Have a separate torrent file for audio, video, and pdf. A separate folder for each author (for books) or subject (for audio/video). And don't zip anything up, since torrent clients allow you to download individual files instead of the entire torrent. If a downloader only wants M,E,S, and has to download the entire torrent, he'd most likely delete everything he didn't want, and therefore, wouldn't seed it all back.

  Also, make sure you stick with whatever naming configuration you decide on. So in a year or so, when you have additions and create a new torrent, people won't have to redownload the older data in order to seed them.

  So if this is the folder structure:

  Mises Books\Rothbard, Murray\Man, Economy, State (Scholars Edition)

  and you add a bunch of new books in a year in and an updated torrent, make sure you still have

  Mises Books\Rothbard, Murray\

  as the folder structure with the same file names for the older books. This way, when you download the new torrent, the client will see you already have those files and won’t redownload them.

  Also, if the database files change often, I wouldn't use torrents. Just zip it up and upload it to megaupload or a similar site.

  
  

  Published: February 21, 2009 9:55 AM

• Mike

  Great post, James R.

  I like Curt's plan for audio/video, but what about all the random individual files? MiscLiterature.torrent?

  Published: February 21, 2009 10:00 AM

• MTB

  Here's how I'd organize the media.

  Have a separate torrent file for audio, video, and pdf. A separate folder in the torrent for each author (for books) or subject (for audio/video). And don't zip anything up, since torrent clients allow you to download individual files instead of the entire torrent. If a downloader only wants M,E,S, and has to download the entire torrent, he'd most likely delete everything he didn't want, and therefore, wouldn't seed it back.

  Also, make sure you stick with whatever naming configuration you decide on. So in a year or so, when you have additions and create a new torrent, people won't have to redownload the older data in order to seed it back.

  So if this is the folder structure:

  Mises Books\Rothbard, Murray\Man, Economy, State (Scholars Edition)

  or

  Mises Audio\Podcasts\12-31-08 Rockwell

  and you add some new books or audio in an updated torrent, make sure you still have

  Mises Books\Rothbard, Murray\

  as the folder structure with the same file names for the older books. This way, when you download the new torrent, the client will see you already have those files and won’t redownload them.

  Also, if the database files change often, I wouldn't use torrents. Just zip it up and upload it to megaupload or a similar site.
  

  Published: February 21, 2009 10:01 AM

• David Veksler

  So who wants to create these torrents? Let me know and I will send you the login information.

  Published: February 21, 2009 10:21 AM

• Mike Byers

  Sites like demoniod provide encompass both a tracker and a site that allows users to find torrents they want and to upload new torrents. The new torrents are created naming the demoniod tracker so that it can coordinate connecting seeders and leechers while a torrent is active. Before the integration of trackers and tracker sites, torrents might be announced via email with the torrent itself included as an attachment. The tracker sites just make it easier to add torrents and find them and download the torrent.

  I don't think this is what you are after. You just want a big torrent that allows bit torrent users to download the entire file set required to run the site, right?

  So you setup the files on a directory and set up a tracker and create a torrent naming that tracker and the files contained in the torrent. Then allow users to download the torrent file from this site.

  As files change, you can create new torrents and users can download that torrent and have their torrent client use their previously downloaded copy of the previous torrent. The bit torrrent content will only download new and changed files (I'm not sure if it will delete the local copy of files not present in the new torrent).

  Published: February 21, 2009 11:03 AM

• Egosumabbas

  Did you get my email?

  Published: February 21, 2009 11:14 AM

• Deefburger

  I'm still downloading media and havn't checked yet, but is there a PDF of the Constitution?
  The Magna Carta?

  Published: February 21, 2009 11:46 AM

• MTB

  David, if you know the naming/folder structure, and have a tracker already setup, I can create the torrent for you.

  Published: February 21, 2009 12:20 PM

• Steven

  I got some experience and enough bandwidth, so I would really like to spread some Austrian liberalism ;)

  Published: February 21, 2009 4:33 PM

• Peter

  Have a separate torrent file for audio, video, and pdf. A separate folder for each author (for books) or subject (for audio/video). And don't zip anything up, since torrent clients allow you to download individual files instead of the entire torrent. If a downloader only wants M,E,S, and has to download the entire torrent, he'd most likely delete everything he didn't want, and therefore, wouldn't seed it all back.

  Eek! No. MES should be a single torrent, by itself. Huge torrents containing 2000 files of which you only want one are a pain - sure, you can select only the one file, but pretty soon you end up with only partially-seeded torrents that are useless to anyone. Some small booklets that are only a few pages can be put together into a single torrent, but generally you should have a torrent for each item.

  Published: February 21, 2009 6:34 PM

• Kevin D. Johnson

  If you need help setting up the server, let me know.

  Published: February 21, 2009 7:12 PM

• Robert C

  Given Peter's compelling argument against a single mega-torrent, I now think that having separate torrents for each item would be better.

  I'd still like to see the availability of one torrent containing all of the literature. Or even a separate torrent for all literature on a single topic (history, ABCT, journals, etc. much like the way these topics are offered in hardcopy sets in the Mises Store) That would make it very easy for me to get all the lit on the site without having to download hundreds of torrent files.

  I think that video and audio should be arranged in small categories, eg. a single torrent for all the video for Mises University 2008, etc. Audio and video should be kept separate. That would make getting complete sets more convenient without leading to the "huge but useless torrent" problem.

  I'm very excited about this. When the seeding project starts, I'll seed Austrian content at high priority. So few people take advantage of my Linuxtracker seeds that it'll be nice to add to the mix with something even more useful.

  Published: February 21, 2009 10:09 PM

• osjak

  I am not a torrent expert, but I do have some experience with torrent trackers. My opinion is this tracker has to be a separate (from the website) machine to keep torrent traffic separated form web traffic. This way the website would keep working as usual even if the tracker server slows down under heavy loads. And it should probably be Linux because you will get many more tracker software options. As fo the webserver for your tracker I would vouch for nGinx - this baby can take a humongous number of requests per second and will outperform anything I know on the same hardware.

  Published: February 22, 2009 1:38 AM

• Scott Baar

  I'm good to seed up to 15 gigs. I'm bigwig in the forums.

  Published: February 22, 2009 2:21 AM

• Paul H

  Kudos James R and Egosumabbas

  I agree, that we need to be clear about the Goal that we are trying to solve with this technology is. If the goal is to just go FOSS with LvMI I'm cool with that, though I don't think it is a generally useful thing for the public in general to download an 80gb torrent file compared to just following http://www.youtube.com/user/misesmedia or other similar services where one can access LvMI media through. (Scribd for pdf's, videolectures.net, academicearth.org for video/audio and maybe some Amazon S3 for off-site backup storaging (I'd donate money with http://www.chipin.com/, if you inform us of the prices))

  Also, since the database with pdf's isn't larger than ~4gb (being the skeleton of LvMI). It could be an idea to separate this for itself as a zip archive in a public Dropbox (http://www.getdropbox.com/) or as the Subversion entrance that was already provided in the previous post http://blog.mises.org/archives/009475.asp

  Published: February 22, 2009 2:27 AM

• Paul H

  Actually Scribd ( http://www.scribd.com/ ) also provides embedding of pdf's across websites. I won't argue if it is prettier or not compared to the design of LvMI, but it provides the possibility for LvMI to not have to store this media either. So in this case it is possible to host all media (pdf (no djvu?),mp3,mpg,aac) on external services or collect them in a Amazon S3 base leaving the LvMI database with only ~200mb of information. Then I would suggest to make two distributions: 1) The FOSS LvMI database (~200mb). Possibly as a torrent, or a public Dropbox ( http://www.getdropbox.com/ ), or a public instance on Github ( http://github.com/ ). 2) A big torrent for all the collection of media of LvMI.

  I guess that the database is updated way more frequently than any of the media, so it that way I think it would make sense to separate the two.

  Published: February 22, 2009 3:16 AM

• David Veksler

  I have created a test torrent at http://mises.org/services/torrents/MultiMedia.torrent

  Published: February 23, 2009 5:27 PM

• Paul

  >60 Gb...if I can identify which bits I've already got, I'll seed those pieces, but One Big Torrent is not the Right Thing.

  Published: February 23, 2009 7:26 PM

• David Veksler

  > I'll seed those pieces, but One Big Torrent is not the Right Thing.

  That depends on whether the purpose of the torrent is to distribute the content to end-users or content publishers. Realistically, I don't have the time to create hundreds of torrents for end-users, so that will wait for a volunteer to step up and do it.

  I do plan to create several smaller torrents with smaller chunks of the site, but they will be > 1GB.
  

  Published: February 23, 2009 8:17 PM

• Paul

  Well, I have just over 1 Gb of that - I mostly transcode and delete the originals to save disk space; the only thing I keep in the original format is PDF, and the books aren't in there....but I'm seeding what I can.

  Published: February 23, 2009 9:26 PM

• cellofellow

  You can't have a torrent file be auto-generated daily or on-demand. For one, that's much to slow. Two, if you change so much as one byte in the source files, the hash in the torrent file with change (well, maybe not one byte but you get the idea). A torrent file is basically a hash and an address for the tracker (and, if I'm not mistaken, web seeds). If you make a new torrent file that is supposed to represent the same-but-slightly-different data, it will be a completely new torrent that will have to build a completely new swarm.

  So, don't go making a script to dynamically build new torrent files that have *everything*, but only use torrents for individual files or for collections that will not change.

  -Josh

  PS: A few months ago I downloaded a small collection of videos from the Mises.org site, and since doing this was a real pain, I figured others would appreciate it and put those files in a torrent. http://isohunt.com/torrent_details/51790881/Economics+in+one+Lesson?tab=summary

  Published: February 23, 2009 11:52 PM

• cellofellow

  You can't have a torrent file be auto-generated daily or on-demand. For one, that's much to slow. Two, if you change so much as one byte in the source files, the hash in the torrent file with change (well, maybe not one byte but you get the idea). A torrent file is basically a hash and an address for the tracker (and, if I'm not mistaken, web seeds). If you make a new torrent file that is supposed to represent the same-but-slightly-different data, it will be a completely new torrent that will have to build a completely new swarm.

  So, don't go making a script to dynamically build new torrent files that have *everything*, but only use torrents for individual files or for collections that will not change.

  -Josh

  PS: A few months ago I downloaded a small collection of videos from the Mises.org site, and since doing this was a real pain, I figured others would appreciate it and put those files in a torrent. http://isohunt.com/torrent_details/51790881/Economics+in+one+Lesson?tab=summary

  Published: February 23, 2009 11:57 PM

• MTB

  I just tried grabbing a few files out of this torrent, and received this error:

  Failure: Torrent deleted or not in pool yet. Go to Demonoid.com and read the FAQ for more info

  I wouldn't use Demonoid as a tracker since it's quasi-private. Maybe the piratebay, or another public one.

  Published: February 26, 2009 7:31 PM

• MTB

  update:

  I woke up this morning and the download was complete.

  Looks like its working just fine.

  Published: February 27, 2009 8:17 AM

• Robert C

  My torrent program shows the country that downloaders are from... the files are going to China! One down, a billion to go.

  Published: February 28, 2009 10:00 PM