| openmosix far from dead, cluster linux without rewriting code? they did it! |
[Jun. 23rd, 2008|03:40 am] |
(02:43:11 AM) The topic for #openmosix is: Welcome to LinuxHive. OpenMosix Re-Visited. check out our docs at http://linuxpmi.org/ , or join the dev team in #linuxpmi
(02:43:25 AM) You are now known as gavin0
(02:45:07 AM) juri_: hio. :)
(02:49:10 AM) juri_: ok! patch 46 commentaried.
(02:52:00 AM) gavin0: oh yeah
(02:52:07 AM) gavin0: linuxmpi.org
(02:52:12 AM) gavin0: name not zippy enuf for me
(02:52:26 AM) juri_: you know, everyone says that. :P
(02:52:31 AM) ***juri_ laughs
(02:52:32 AM) gavin0: pangalacticprocesbalster.org
(02:52:48 AM) ***juri_ laughs again. :)
(02:52:50 AM) gavin0: zaphodprocessbrox.org
(02:53:28 AM) juri_: i think i'll keep linuxpmi.org, but thats just for the kernel-module part of what used to be openmosix. everything else needs to file itsself under a new name.
(02:53:35 AM) gavin0: teamix.org
(02:53:41 AM) gavin0: makes all hardware team up?
(02:53:55 AM) juri_: and for that to happen, first someone else needs to define what 'everything else' is, and take charge of it. :)
(02:54:01 AM) gavin0: uh what is there aside from a kernel module!
(02:54:09 AM) juri_: quite a lot, actually.
(02:54:12 AM) gavin0: oh?
(02:54:19 AM) gavin0: like what? monitors?
(02:54:22 AM) gavin0: er
(02:54:38 AM) juri_: yepyep. process scheduling algorithms. the code that tracks what nodes belong to your cluster. the stuff to actually tell the kernel when to move a job.
(02:54:46 AM) juri_: this is now OUTSIDE of the kernel's domain.
(02:54:59 AM) juri_: and, as such, not my problem. :)
(02:55:12 AM) gavin0: uh whos is it?
(02:55:31 AM) juri_: i've got enough on my plate. if someone else wants to take up the other half of the flag, great. if not? i'll get around to it around december or so. :P
(02:56:01 AM) juri_: so far, i'm the only one contributing, other than an ocasional admin-type-person.
(02:56:23 AM) gavin0: Id love to use openmosix soon......problem is web stuff is only thing I think I can sell.....and web crap ususally demands mutithreading for performance......Im learning aolserver+tcl+postgresql now in a vain attept to learn something that will SCALE, unlike wordpress and php
(02:56:51 AM) juri_: mmh. openmosix has had more scalable versions in the past.
(02:57:15 AM) juri_: the version with checkpointing, migrating shared memory, and distributed sockets was beautiful.
(02:57:18 AM) gavin0: If I can get it to work.......I will be able to host dynamic sites on 5% of hardware of my competitors
(02:57:36 AM) gavin0: checkpointing?
(02:57:44 AM) gavin0: so you do this all in c?
(02:58:14 AM) juri_: checkpointing made it where when a node crashed, no 'state' was lost. the processes were just re-started elsewhere, without losing much of their place.
(02:58:29 AM) gavin0: Im not skilled in c yet........although I do have a k+r book sitting here beside my haskell lisp and smalltalk books, none of whihc I know yet....Im so damn loser, haven't read books i own
(02:58:32 AM) juri_: yepyep. linuxpmi is all C. the userspace stuff? i don't care about(yet). :)
(02:58:55 AM) juri_: i have lots of books on my reading list. i'm a bit busy with openmosix however.
(02:59:01 AM) gavin0: you know whats interesting? linux MPI is some other clustering thiny
(02:59:32 AM) gavin0: maybe I should quit screwing with all thsi stuff and learn c and help you out
(02:59:48 AM) juri_: rule #1: take care of yourself first.
(03:00:02 AM) gavin0: if I could figure out hwoto make tiny processes......then I would have a huge win with linuxpmi because add a box scaling is sick
(03:00:20 AM) juri_: if you've already got one problem on your hands (scaling web programs), adding another problem won't make your situation any better. :)
(03:00:55 AM) juri_: now, it depends how your webserver/programs are scaled.
(03:00:56 AM) gavin0: I like too many technologies.......I really like forth and lisp....and think haskell is awesoem as well...plus smalltalk too
(03:01:13 AM) gavin0: aolserver is c+tcl
(03:01:25 AM) gavin0: so if I can learn tcl......aolserver will fall into place
(03:01:28 AM) juri_: so, no seperate processes to migrate then!
(03:01:34 AM) gavin0: yeah
(03:01:44 AM) gavin0: aolserver is multithreaded sinc 1995
(03:01:45 AM) juri_: making openmosix the wrong tool, until it gets MIGSHM and MIGSOCKET.
(03:01:50 AM) gavin0: yeah
(03:02:11 AM) gavin0: but openmosix+fascgi, if it could be made to work.....would eb sick
(03:02:14 AM) gavin0: be-
(03:02:39 AM) gavin0: since all the new webservers liek lighttpd and nginx just hand off to a backend running processes prefork in FASTcgi
(03:02:46 AM) juri_: well then, if you want to do something to help both yourself, and the project in general, try getting a openmosix 2.4 with migshm running, then try finding enough information to get migsocket going.
(03:02:53 AM) gavin0: those cgi if they can move around would be instantly scalalble
(03:03:08 AM) juri_: i'm not sure how fastcgi does its thing. :)
(03:03:17 AM) gavin0: prespawns processes
(03:03:19 AM) juri_: which is funny, since i'm using it...
(03:03:33 AM) gavin0: then event based webserver such as nginx just throws requests at em
(03:03:37 AM) juri_: so, if they're seperate processes, what methods of IPC are they using?
(03:03:52 AM) gavin0: :) no idea, i can go look it up...
(03:04:02 AM) gavin0: linux uses pipes and signals
(03:04:07 AM) gavin0: Id guess ignals?
(03:04:11 AM) juri_: i think that would fall under 'helping yourself'.
(03:04:25 AM) juri_: signals are single-digit numbers, with assigned reasons. thats not a request.
(03:04:29 AM) gavin0: migsocket?
(03:04:34 AM) juri_: i'd guess pipes, but it might be using shared memory.
(03:04:57 AM) juri_: MIGSOCKET is socket migration code. written by one guy one time. i have his paper, but have yet to try and convert it into usable code.
(03:05:03 AM) gavin0: I think they are long running cgi programs pre spawned, they jsut inherit variables from new request
(03:05:27 AM) gavin0: then variables are cleaned out, and process is reused...
(03:05:48 AM) juri_: right. its that IPC method i'm interested in. if fastcgi worked on openmosix, that would be handy.
(03:09:31 AM) gavin0: Instead of using operating system environment variables and pipes, the FastCGI protocol multiplexes the environment information, standard input, output and error over a single full-duplex connection. This allows FastCGI programs to run on remote machines, using TCP connections between the Web server and the FastCGI application.
(03:09:50 AM) juri_: mmh. so sockets.
(03:09:53 AM) juri_: -EFAIL
(03:10:15 AM) gavin0: epic fail?
(03:10:17 AM) juri_: fastcgi's using sockets to do its communication. MIGSOCKET is not even a source file on my disk yet.
(03:11:19 AM) juri_: the ironic part is for heavy CGIs, the old CGI method worked on openmosix. CGIs just used pipes, which worked with the 2.4 branch.
(03:11:34 AM) juri_: its just that CGIs usually don't live long enough to get migrated away.
(03:11:59 AM) juri_: and the ones that do live that long are doing things like database access.. again, over sockets. doom. ;)
(03:12:23 AM) zoli2k [n=zoltan@ufv063-33.science.upjs.sk] entered the room.
(03:13:15 AM) juri_: so, it sounds like one way or the other, you're going to need the migrating socket work of ethan bowker.
(03:13:27 AM) juri_: which, as far as i know, never made it into a functioning version.
(03:13:53 AM) juri_: so, check out krex.k-state.edu/dspace/handle/2097/536
(03:14:31 AM) gavin0: ah crap
(03:14:35 AM) juri_: and try examining a path of getting that into SOME version of openmosix. 2.4 based preferably, so you can get something functional, whilst i keep pluging away at 2.6.
(03:16:27 AM) gavin0: woa
(03:16:35 AM) gavin0: smart guy it seems
(03:21:34 AM) juri_: aparently, the maask group also wrote an implimentation of socket migration.
(03:21:44 AM) ***juri_ hunts down one of the authors.
(03:21:47 AM) gavin0: hm
(03:21:53 AM) gavin0: sounds like a lot of work
(03:21:58 AM) gavin0: migrating all that stuff
(03:22:02 AM) gavin0: hmm
(03:22:12 AM) juri_: its a huge project. i should not be doing this alone. :)
(03:22:16 AM) gavin0: see load balancing si what I use as a linux admin
(03:22:28 AM) gavin0: why did moshe n all them stop?
(03:22:39 AM) gavin0: are beowulfs better way to cluster or something?
(03:22:42 AM) juri_: thats a complicated question.
(03:22:43 AM) juri_: nope.
(03:22:53 AM) juri_: so far, openmosix is the bleeding edge, and the holy grail.
(03:23:06 AM) gavin0: why do beowulfs suck?
(03:23:14 AM) juri_: i think it had more to do with the fact that when full system virtualization came along, massive money tore the group apart.
(03:23:23 AM) juri_: because they're batch computing. :)
(03:23:40 AM) gavin0: whats batch computing?
(03:24:28 AM) juri_: to run a program on beowulf, you must re-write the program parallel-like, where it uses the beowulf API, then launch it everywhere.
(03:24:41 AM) juri_: OM is transparent. the processes don't need a re-write.
(03:25:29 AM) gavin0: so processes dont even know
(03:25:33 AM) juri_: right.
(03:25:34 AM) gavin0: new modes just start helping
(03:25:38 AM) gavin0: nodes---
(03:25:44 AM) juri_: thats part of what makes it so complicated. ;)
(03:25:48 AM) gavin0: hm
(03:26:06 AM) gavin0: hmmmmm
(03:26:11 AM) gavin0: that is awesome
(03:26:32 AM) gavin0: now if fastcgi just has processing sitting there prespawned
(03:26:53 AM) gavin0: like littel ducks in a poind
(03:26:56 AM) gavin0: pond
(03:27:05 AM) gavin0: then I wonder why it needs sockets
(03:27:11 AM) gavin0: the info flow accross the socket?
(03:27:13 AM) gavin0: hm
(03:27:17 AM) juri_: thats just the IPC method they chose.
(03:27:21 AM) gavin0: have you seen rubyqueue?
(03:27:31 AM) juri_: they likely could have done it with pipes.
(03:27:43 AM) juri_: nope. not a web-development type person anymore. :)
(03:27:58 AM) gavin0: http://raa.ruby-lang.org/project/rq/
(03:28:21 AM) gavin0: this is one thing along with openMPI that always seemed k00l
(03:28:30 AM) gavin0: rq is not webby
(03:28:57 AM) juri_: you'll pardon me if i'm not very interested. i'm a C developer with the world on my shoulders, right now. :)
(03:29:00 AM) gavin0: and one other thing I saw was plan9 os, where someone can export a cpu or memory as NFS exports a directory
(03:29:03 AM) gavin0: ok
(03:29:06 AM) gavin0: sorry
(03:29:10 AM) juri_: my applications are already written in C. :)
(03:29:13 AM) gavin0: you are the man
(03:29:21 AM) juri_: Nop! :P
(03:29:28 AM) gavin0: c c c !!
(03:29:40 AM) ***juri_ is julia elizabeth longtin. no man here. :P
(03:29:41 AM) gavin0: did you learn c from k+r?
(03:29:47 AM) gavin0: oops Im sorry
(03:29:50 AM) gavin0: :)
(03:30:00 AM) juri_: actually, i learned it from a microsoft C programming book, back when they wrote good books.
(03:30:09 AM) juri_: i learned C before i touched linux.
(03:30:11 AM) gavin0: long time ago ;)
(03:30:16 AM) gavin0: har har
(03:30:21 AM) gavin0: wow microsoft
(03:30:27 AM) juri_: yepyep. 94ish.
(03:30:32 AM) gavin0: I thought they only allowed peple to do vb
(03:30:46 AM) juri_: this predated VB being big. :)
(03:30:48 AM) gavin0: c can be a huge win for me
(03:30:57 AM) juri_: hell, it may have been as early as 90.
(03:31:02 AM) gavin0: since the afrementioned tcl works well with it
(03:31:04 AM) gavin0: 90 wow
(03:31:12 AM) gavin0: amazing thats 18 years ago
(03:31:15 AM) gavin0: how surreal
(03:31:21 AM) juri_: don't remind me. i'm old. :)
(03:31:29 AM) gavin0: someone born in 90 is 18, can vote
(03:31:33 AM) gavin0: Im 35
(03:31:37 AM) gavin0: feel 22
(03:31:54 AM) gavin0: Im like hey where did this extra weigth coem from and why can I not run 5 miles?
(03:32:05 AM) gavin0: lol
(03:32:16 AM) gavin0: <--sore from working out
(03:32:46 AM) gavin0: ever use linux desktop +openmosix?
(03:33:09 AM) juri_: long ago. :)
(03:33:39 AM) juri_: since i got involved in the 2.6 branch, i've been focused, like a lazer. can't think about much else, it steals cycles from what i'm trying to do.
(03:34:04 AM) juri_: ok, i've hunted down one of the authors of the MAASK group's MIGSHM and MIGSOCKET code, and sent him an email. heres hoping he has it.
(03:35:13 AM) gavin0: :)
(03:35:36 AM) gavin0: how much of an adjsut ment from using a linux distro like redhat si using openmosix?
(03:35:48 AM) gavin0: does it install from a cd?
(03:36:02 AM) gavin0: I feel like getting it up on a pair of boxen I have
(03:36:03 AM) juri_: its just a kernel replacement.
(03:36:16 AM) gavin0: I use archlinux
(03:36:22 AM) juri_: in theory, you just replace your kernel with an openmosix kernel, and supporting userspace tools, and it starts 'doing stuff'.
(03:36:35 AM) gavin0: if I recompile the kernel to linuxpmi will it explode my userspace stuff?
(03:36:39 AM) juri_: unfortunately, the 2.6 version of openmosix was never completed.
(03:37:01 AM) gavin0: ok so stick to 2.4 until you work your magic
(03:37:04 AM) juri_: so, in a short, no, the kernel still works with all our stuff in, but when you try using it, things explode violently. :)
(03:38:01 AM) juri_: right. use a 2.4 until i've got things working.
(03:38:10 AM) gavin0: k
(03:38:19 AM) gavin0: when I install, what will I see?
(03:38:24 AM) gavin0: does it have an installer?
(03:38:28 AM) juri_: mind you, if you use a 2.4, that will make you our relivant expert on 2.4 openmosix. its been years since i touched it. :)
(03:38:38 AM) gavin0: lol
(03:38:43 AM) juri_: nopenope. you just patch your kernel, and re-install it.
(03:38:59 AM) gavin0: so its liek a kernel compile
(03:39:02 AM) gavin0: ok
(03:39:08 AM) gavin0: so I leave my shell etc alone?
(03:39:14 AM) juri_: exactly.
(03:39:18 AM) gavin0: what userspace stuff needs to be added so no explosion?
(03:39:25 AM) gavin0: anything?
(03:39:33 AM) juri_: you have some programs to install to manage the cluster, and a configuration file to update.
(03:39:47 AM) gavin0: trivial for a linux admin such as me
(03:39:51 AM) gavin0: bah
(03:39:54 AM) gavin0: a midnight run!
(03:40:24 AM) gavin0: now where can I get a 3rd box, since I dont wana trash my windows machine
(03:40:26 AM) gavin0: har har |
|
|