[ Posted by kevin
Mon, 13 Jun 2005 16:35:10 GMT ]
When I first wrote allpoetry, I optimized it so that no query was not fully indexed. This makes for big indexes, and very fast performance. But I’m starting to realize that other’s instead add database servers instead to distribute out the ‘slow’ load.
Part of the whole problem is that disks are very slow. Since ram is cheap these days, I’m more interested in a in-memory database, especially if that can cut down on the index size.
Allpoetry is about 4 GB of database now, a little over half indexes.
Mysql Cluster is one option, but requires 2*Database size+ some memory, and at least two servers. But it scales nicely. But it isn’t that fast. I think it’s lame to need so much memory. An object-oriented database like madeline (based off a similar java system), doesn’t have any need of doubling the data storage? But perhaps mysql’s statistics includes the indexes?
Another option I’ve been obsessing about lately is the idea of running mysql on a ramdrive. It should be pretty straigtforward:
- Create a ramdisk on startup
- Copy the mysql database from /db/allpoetry to the ramdrive
- Symlink a directory under /lib/mysql to the new area
- Make the binary update log on the HD still
- Have a cron job to apply the update log to the disk-version of the database every 30 minutes or whatever.
- Replicate and repeat as normal.
A device like this could help make this much easier:
http://www.tomshardware.com/hardnews/20050601_115506.html
Any ideas or suggestions?
Technorati Tags: databases, mysql
Posted in Coding | 2 comments
[ Posted by kevin
Tue, 07 Jun 2005 04:25:58 GMT ]
You ever notice, how some days, you sit down and work and work, but at the end of the day, can’t say what you did? But you really were working hard!
I’ve been reading the wildly popular “Getting Things Done: The Art of Stress-Free Productivity” (David Allen) lately, and while I’m a pretty organized person, some of it makes sense to me. I definitely plan to try having more folders to put things. Mostly I’ve solved that problem already though by just having a ‘todo’ list (yes, on paper gasp) in front of me, and whenever something comes up while I’m in the middle of something, I add it to the end instead of doing it right then. I get distracted a lot, I was A-D-D as a child.
I think the more important realization, though, is there is just a bunch of junk in life that you really have to do before you can get work done. And on a new computer, that’s at least a few days work. For me, a whole day was more time than I could imagine, so I was really frustrated that it took me longer than that. Plus playing with all the new changes in OS X, etc. Getting accustomed to the environment, and staying a fairly up-to-date in computerisms takes a lot of time.
So relax, stop trying to hard, and know that forward progress is more important than manic fits of seems-like-productivity.
Technorati Tags: getting things done
Posted in Coding | 1 comment
[ Posted by kevin
Tue, 07 Jun 2005 04:19:17 GMT ]
You might have noticed, but I run a big poetry website called allpoetry.com. For the first few years I ran nightly analysis on the logfiles, then hourly as the site got bigger, but it kept slowing things down.
And I never looked at it. So I disabled it, and I just clean up the log by hand every few months once it reaches a few gigabytes.
The important thing with statistics is figuring out what questions you want to be answering. To me that’s one of the big problems with research sometimes – rummaging through answers to questions we don’t have, and might not ever.
Anyways, I’m playing around with me-driven logfile analysis of the last 100,000 rows (just using ‘tail’ whenever I want to look at the stats. Not quite as useful as historical stats, but faster.
One of my big problems also was that since it’s a dynamic site, many pages are unique. I really want a tool that will ‘stem’ my pages, so /poem/14441?reply=yes will just be /poem. It’s more useful to see that there were 20,000 hits on poems, rather than 20 hits on poem number 14441.
I’m sure others have confronted this problem before; if you have any ideas, let me know!
Technorati Tags: logs
Posted in Coding | no comments
[ Posted by kevin
Thu, 26 May 2005 22:07:00 GMT ]
It’s time to put my money where my mouth is. I spent way more time on my computer than in my car, so justifying spending nearly $6,000 on a computer shouldn’t be that hard.
Several hours of hair tearing and palmy sweating, the online purchase is complete! And now, a week later, I’m sitting in front of the fastest computer in the world, with the biggest monitor in the world. It’s good to be a geek!
Just in case you don’t know what ‘the fastest computer in the world’ means, it’s a dual 2.7 ghz Powermac g5, with a 30” monitor. Man, that’s big. And fast.
Took me most of a day getting the various programs I like reinstalled, and moving things over without the ‘migration program’ (I hate clutter, part of getting a new machine is starting mostly afresh).
Posted in Coding | no comments | 266 trackbacks
[ Posted by kevin
Fri, 10 Dec 2004 21:35:00 GMT ]
In case you didn’t notice, I’ve been a big fan of rubyonrails lately. Its a slick new platform for web applications, and just happens to be written in ruby, which I recently decided was the most elegant language ever written. I can’t stand writing perl anymore. That says alot.
Anyways, there’s this killer editing program for ruby thats OSX only (textmate) and I wanted it so badly I bought it. How do you get a mac application to run on a windows computer? PearPC you go out and buy a new ibook laptop :)
Under $1000 and so beautiful I want to cry, I won’t lie and say I was used to it immediately, but it has been a fairly easy transition. A few days with going haywire and filling up my disk with fun applications and customizing everything, but that definately goes in the ‘playing’ rather than ‘working’ category. Everything about this beast is beautiful. Did I mention it hasn’t crashed since I bought it?
Posted in Coding | no comments | 1 trackback
[ Posted by kevin
Mon, 15 Nov 2004 21:39:00 GMT ]
The last 6 months have been a blur. I’ve:
- adopted css
- learned ruby
- decided I really hate perl, despite having big one of it’s largest supporters in the past
- become obsessed with rubyonrails
- learned design (by following 37signals websites across the net, among others).
I’m tired! And my brain is somewhat fried. I’ve lost some of my confidence I had before, but I’m very confident that things are moving in the right direction. Having so much change so fast is difficult (i.e. I really thought THAT code was BEAUTIFUL before? Argh!)
And I’m realizing that none of these things I’ve learned are a silver bullet (to pull a term from the Mythical Man Month). Designing a complex site is still a pain in the butt. And ‘porting’ code I’ve written over the last 5 years is a huge, painful project.
Still, I’m excited for the future. I just need to catch my breath, and not be so hard on myself when things take longer than I want. Optimism can be a curse when you end up pushing yourself too hard!
Posted in Coding | 10 comments | 6 trackbacks
