The advantage about discussing this a la opensource is that it sends strong signals to would-be crackers, as well as reminding fully fledged crackers that the best programmers work to CATCH THEM not help them. Believe me I can write the best brute-forcers in the world, but for now I haven't bothered.

Here's some planning work for a new anti-hack system I am preparing for my server:


OBJECT 1: netstat monitor...
a shell script, launched by the crontab (or rather launched from a scheduler the crontab is set to launch at regular intervals, so that I don't need to recrontab anything, I just add the new commands to the scheduler), will now log the output of netstat on a regular basis, send it to a perl script which will dissect the data, examine it to see if any ssh connections are alive which shouldn't be alive. it will also look for one or two other signature characteristics of hacker accesses I've seen happen and taken notes from

this object's purpose is to constantly patrol the access points - it won't just monitor ssh accesses, it will look for other potential break-in points

the perl script it runs will then determine whether or not there is any reason to imagine a hack is in progress. if one is in progress, or suspected, it will not only send me an email, but it will alter my own view of my domains such that it will not be very long before i end up being given an alert by my machine, no matter where on my server i go or with what domain... as well as this, if i had money in the bank, i could easily get it to send me a text message alert, direct to my mobile phone which is always with me - thus i'd have an instant alert of the hack

more importantly, i'd have collected the data showing when it was found to be going on, so i no longer only have before me the immediate data, when pursuing a hacker

naturally it is not at all hard for the shell script to find the process ids of the hacker's entry and usage... and with the help of, at most, a bit of perl, or just some versatile shell, it can automatically terminate any such process whether or not i am able to get to a machine and handle the bugger myself



some crucial commands to be adept with when pursuing security trouble and tracking issues on your linux server...

top...
(in some cases you may find that server usage patterns are a good way to find hack or indeed spam activity... probably in more cases than not - certain apps running in certain ways, for certain abnormal lengths of time - you are just looking for abnormalities, after all, for that is the biggest clue possible about breaches and nefarious activity)


netstat
obviously - this helps you see who's using what, very swiftly

who
helps you get a clear definition of the hacker's identity

ps -e

and just

ps


then of course you need to master kill -9

and be careful with kill. always be careful with anything relating to the concept of kill. isn't that obvious? true programming wizardry comes from common sense, not books.


Okay, what else could you do to build and maintain strong anti-hack capabilities?

Well Tripy and others would wisely suggest you subscribe to mainstream systems of various kinds, add stuff to your server which others have written for blocking hack and spam

The disadvantages of relying on that include:

1. Using stuff which is documented, out in the public domain and itself a target of other/new hacks
2. Blocking out things without knowing precisely what - so if there are errors and vital business/warranted usage is messed up and you don't know in advance - creating it all yourself means you can work out what can go wrong and when miscarriages of justice may crop up

the advantages don't need explaining. they are obvious.

So apart from a basic "patrol" system, what else can you use? Patrol-alert/terminate... that's object 1, so object 2 is...



OBJECT 2: invisible man viewer
okay, the next important thing is a script, blatantly a shell script, probably no need for perl if you can handle slightly tricky commands like sed, which regularly scours the entire machine, every nook and cranny, for files named with a . at the start - invisble files, which need ls -a rather than just ls to make sure you pick them up in directory listings. on top of this a sort of special map, of your own, mapping all files throughout your system, and every hour check to see if anything has been added in areas you cannot account for - over time you can easily learn to account for ordinary automatically generated information produced by the ever-entropic ever-moving system you run, and anything created by your own mechanisms, ie your own programming, you'll easily be able to account for, so it isn't hard to write shell and no doubt perl to help you know, every hour of every day, whether or not anything is on your server which can be "suspected" of being a breach, a trojan, anything malignant at all... such items can be quarantined in certain ways (eg made totally unexecutable until you are able to verify their innocence in person)


OBJECT 3: history file
I think for now the only other thing I can think of is the concept of a "history file" - a permanent record of what breaches have occurred over time,


and beyond objects 1 to 3...

the first obvious thing is don't be "free" and open with all your info - if you want to access your own info every day, in private, DO put passwords and walls around it, don't be lazy.

accept that what doesn't even have passwords is not secure. secrecy is no help against hackers. hackers can watch your private life... so your security-free pages may turn out to be found out very easily - even the least sensitive info, if it is supposed to be private, should be secure and cordoned off.


OBJECT 4: obvious now...

email monitor... a shell script watches the sendmail logs closely and with the help of a perl script or merely sensible seds and sorts and uniqs and cuts, checks for any non-standard email behaviour - on most servers I suspect it would be very easy to spot breaches and hacks and even spam that is done by non hackers. etc.

so an email monitor is vital, since very frequently that's what the hacker is aiming to gain control of. obviously in order to send billions of messages out without being "traceable" - you get the blame for what they do.

Problems:
A good hacker will use a currently running executable to do whatever they need. So, netstat and PS would show nothing abnormal. top might show greater than normal cpu/memory usage for the process though. who is only going to show which user they are logged in as. It'll only identify people who already have access to you system.
All good hackers clean up history files so, that'll only be good at watching the amateurs.

....or, you could just run tripwire, AIDE, or Samhain

If you're worried about mail server exploits, get rid of postfix or sendmail (depending on your install) and just run Apache James. It's a heck of a lot more secure and doesn't need to run with superuser privileges. Run it listening on higher port numbers and use iptables to route traffic from the standard mail ports.

Originally Posted by willcode4beer

Problems:
A good hacker will use a currently running executable to do whatever they need. So, netstat and PS would show nothing abnormal. top might show greater than normal cpu/memory usage for the process though. who is only going to show which user they are logged in as. It'll only identify people who already have access to you system.
All good hackers clean up history files so, that'll only be good at watching the amateurs.

....or, you could just run tripwire, AIDE, or Samhain

If you're worried about mail server exploits, get rid of postfix or sendmail (depending on your install) and just run Apache James. It's a heck of a lot more secure and doesn't need to run with superuser privileges. Run it listening on higher port numbers and use iptables to route traffic from the standard mail ports.

netstat showed me that a foreign IP was accessing my machine via ssh (other than my own, of course) (plus the IP in question was preceded by a bunch of FFFs)! That was a major giveaway, and a major breach. but otherwise it's true I wouldn't have gained anything from netstatting.

top is what got me to suspect something was amiss - a whole bunch of perl processes running in ways none of my perl runs - accruing many more minutes of execution time than anything i run myself.

thanks for all your tips and method advice. it all sounds good to me.

my most useful anti-hack idea, as far as I can see, is a bunch of shell scripts and perl scripts whose main job is to monitor any change to my server every hour (or less if poss), primarily seeking to inform me of any new files that are created, above all scanning for invisible ones (i.e. starting with a dot) but obviously examining all visible ones too - to me, as a self-taught server person, that's the absolute best, at this time, I can do... once, years ago, someone put invisible files on my server and created a "fake e-bay" site, for stealing credit cards from punters. to be sure that you can easily catch something like that, I'm sure a system of monitoring new files would work.

nonetheless, I'll go over your tips and try to learn how to implement everything you've mentioned. cheers.

A good way for me to start building my all new anti-hack system (I absolutely must make my own security, I've decided, and it must be the best I can possibly do - I don't mean I'm unhappy to use other security systems, oh no - I just mean that the final line of defence between me and the baddies is always going to be my own arsenal of e-weaponry) would be to make a list of every possible way a hacker can get in - ie every possible door they could break in by (or window), not every possible method which, for now, is slightly less important.

I think that making history logs of activity is quite safe as long as those logs are cleverly buried far out of sight of an average hacker. The recent hack of my server showed me that even hackers capable of breaching ssh (you'll agree that's a very hard thing to do - and it didn't appear to be piggybacked in any way, because the hacker even broke in when I was totally not logged in) are too careless, more often than not, to eradicate their trail properly - I found traces of failed activity in the mail logs, I caught their ip addresses on netstat, I saw their overloaded program executions on top... so all their intelligence when it came to smashing down the safe door was spoiled by the fact that they smiled into the cctv and left a forwarding address!

Anyway, what I want to know from all the good security people here is a full list of possible doorways the muthas would try to breach - here's my own lay shortlist - I'm afraid I have a lot to finish learning about Linux - after the custom of skywalker. So this list is massively insufficient and part of it may even be wrong:


1. ssh
2. ftp
3. http


that's it - that's the only places I'd imagine they'd be trying to get in, and yet I know that there are a few more at the very least. So let's do it - can you help me with the planning and rough work as I build my strongarm security solution?

I will happily "let" others help me and join me in my development if they want - a particularly talented programmer in the netherlands is a very good friend of mine from youth, so I'm not alone on this - although I haven't told him about it yet he'll be very excited. Like me, programming is his life.

The threat is getting very serious, so there's a LOT of money in good security. Other people in my retinue of associates have volunteered to help put the organisational structure (and sofas) behind me such that I can, once I have indeed mastered this field (a good solid year of hard e-labour I think), launch a company which sells solid security to people across the city of London (and then, when a basic clientelle is established, all the world's capitals).

I'm very glad that a few months ago I did an in depth study of how a hacker operates, and wrote out a lot of pages about how you can break into places. It was only a starter really - the main course is now approaching, but that work was crucial - without it I wouldn't have had a broad idea about what I was really facing and I would not have managed to stay so calm, I don't think.

All hands on deck! In our time of success, the "liberal" electronic world of the "western hemisphere" is massively under attack, both from the enemies in our own hemisphere and indeed from their enemies, who manage to encompass most of the entire other hemisphere!



another idea for security objects:

a quick gander at tripwire above led me to read something which brought this idea to my mind -

Every single change to a key system file would have to be preceded by a special code sequence, known only to the systems administrator, obviously programmed using the highest forms of encryption possible within the machine.

Then any time anything tried to change such a file without the administrator's "uber" code sequence, as it would thus be, the server should deny that change. of course i am easily equipped to detect a breach just after it's happened, but the real key is learning how to prevent the action, how to do the monitoring before the action...

no doubt it's not that hard

then again... how in Hilary Clinton do you get it to demand some kind of input from you if you want to nano or pico or joe or vi or any other textedit doohickey a file? if you already have root passwords and everything, it will just let you - so i'll have to actually understand how to interfere with any and all texteditors, and even then what if the mutha makes their own bespoke texteditor and sidesteps my little catch. then again they'd need to install key system files in order to put such an editor onto my machine, so it'd probably be okay. however, how in Melanie Sykes would you actually put maximum security on a file like that - where would you go into the system and **** with the code? I'll get back to my reading now. The plan is to read the entire Oreilly linux book and learn its contents very completely before finally taking the Blue Camel perl book and doing the same with that - the reason I have to do it in this order is that to learn everything about perl you already need to know most things about Linux, possibly all things. A lot of "social engineering"* in perl requires that you know a hell of a lot about Linux and Apache.

What is the best next thing to learn after those two for getting firmly into network security?


by the way, where should I start if I want to start learning how to make daemons and other important linux objects? presumably monitoring every act before it happens to match it against a list of system files requires knowing something about how each command is processed after being issued by human (or human impersonator/intermediary) to machine



* "social engineering" - title of chapter 6 in 2nd edition {covers perl 5} 'blue camel' book, ie Programming Perl, O'Reilly, Wall, Christiansen, Schwartz

If you're worried about mail server exploits, get rid of postfix or sendmail (depending on your install) and just run Apache James.

I reckon I'm just going to switch off the whole mail server. I don't really use it for anything important. My robot tells me how much money I've made regularly. Other than that it doesn't even do anything. It's a pointless temptation to put there before cracker-spammer people. I can check the web to see how much money I've made. I changed my email and never even bothered telling my robot about it - so I don't even get its emails any more. Kind of wise to switch it off - I'm gonna look it up and do it now (it's not something I've ever had to do from the commandline before).

Moments from choosing between seeing what happens if I kill sendmail, or looking up how to just uninstall it entirely, or in some devious way just moving /usr/sbin/sendmail to some other place for safekeeping and letting the server just misfire every time i t sends an email, so that I can still catch any hackers at work and see what they were trying to do!

And then I realised I can't - because one of my commercial systems does actually involve getting a website to send order requests out to an email address.

I suppose I'll have to just upgrade that first - i.e. just send the orders to a weblog instead. Then I really can shut down sendmail. Does anyone know what the most sensible way to do that is? Maybe this one...

http://www.qmailrocks.org/remove_rh.htm



also... another brilliant anti-hack device is a hardware security object - for example you could have a machine that is so unusual it is hard to hack, eg the good old apple macintosh has always enjoyed far higher "natural" security against mainstream hackers since the majority learn about either microsoft or linux - but mac's linux/unix capabilities are of course a weakness there.

having a warped o.s. that has gone wrong and can't install any applications... can be quite a clever security device. there's lots of ways to be clever about how you prevent unwanted activity on your machines and networks. but right now I can't afford to buy a new server and then partially cripple it!

I've changed ssh to only work from one machine now, but even after blocking china and korea with IP deny, one chinese user still established ssh, plus someone in chile and someone in indonesia

I have banned all those countries with IP deny - does anyone know if ipdeny means zero access at all - how come i got one from china anyway - when rebooting do you need to run the ip deny code again?


Now that ssh is successfully limited to one machine and no one can break into it, we'll find out whether those breaches really are straightforward accesses via ssh, or whether something else, fishy, is going on.

Apparently I will still easily catch data of these breach-merchants as they come by. There's apparently enough "amateurs" around to help find the major security holes!

RULE ONE OF SECURITY

I have found out in a very in your face way that the most important rule of protecting servers is this:

KEEP YOUR WHOLE OPERATING SYSTEM COMPLETELY UPDATED

and the second rule of security, which is vital for making sure that rule one is adhered to...

SCHEDULE REGULAR DOWNTIME: MOST IDEALLY, RUN A SECONDARY SERVER AS A BACKUP AND SWITCH OVER TO IT WHEN YOU ARE UPDATING AND RESECURING YOUR MAIN SERVER

and rule three follows from that - constantly resecure your server (i.e. in every way you can think of)

For ssh, consider only allowing connection with a certificate (ie no username/password connection at all)

Regarding mail, you don't actually need a mail server running to send mail. It's really just a matter of connecting directly to the recipient's mail server to send the mail.

FTP? just turn it off and/or uninstall it. Use SCP to transfer files instead (use the cert that you use for SSH).

Other means of getting in your system? Use netstat to see what ports the server is listening on.

Originally Posted by willcode4beer

For ssh, consider only allowing connection with a certificate (ie no username/password connection at all)

Did that now. And removed the root ssh. And updated openSSH to one without known vulnerabilities. And many hours have passed without any further breaches. Plus I have banned access from numerous countries which provide me no commercial activity but plenty of hackers and spammers.

[[there's some really funky lightning right now over the city of london - i.e. the financial quarter]]

Originally Posted by willcode4beer

Regarding mail, you don't actually need a mail server running to send mail. It's really just a matter of connecting directly to the recipient's mail server to send the mail.

I am due to switch off all mail capability very shortly, although for now no one can break in anyway, not in the way they were doing before. Nonetheless, just gotta go sort out one cgi script and get it to write the output to a log or sql or both, rather than emailing me. I hate email anyway, I believe there are more evolved ways to communicate. Sooner or later I'll just send people an app if they wanna be able to talk to me. (Deep down, I like making little guis, when I have the time, but network language work is so overwhelmingly necessary all day every day for one reason or other).

Originally Posted by willcode4beer

FTP? just turn it off and/or uninstall it. Use SCP to transfer files instead (use the cert that you use for SSH).

What's SCP? Is it highly accessible to the consumer/lay-user? Even if I want to use SCP for myself, I think I should probably just learn to massively secure FTP rather than cut it out.

Activists and other progressive people (eg techies) share vital info with me that way, and I think they're probably going to be more comfortable with something traditional like ftp, by and large. Also musicians and music industry people have to use ftp for providing me with their goods.

Originally Posted by willcode4beer

Other means of getting in your system? Use netstat to see what ports the server is listening on.

Will do, cheers.


[[Lots of crazy lightning over London today]]