best log checker

Tags:

Debian User

I'm trying to find a good log checker.

Basically, I want it to report anything that I don't tell it to ignore.

I've tried logcheck first and when I couldn't get it to do what I want I
tried logwatch. It has an ignore file that it says to just cut and
paste what you want to ignore. I do that and it doesn't ignore it.
Some docs mention that its all based on regular expressions so I tried
enclosing the lines in quotes to no avial.

I do neither perl nor RE: they're both too cryptic. I guess I'll never
be a true *N*X weenie.

I _like_ most of what logwatch does, like telling me how many times a
login happened, especially failed ones. I just don't like to have to
pour through all the bootup lines every day.

If I don't find anything else, I'll disable this portion of logwatch and
go back to my brute-force python script for parsing syslog.

I'm running Etch amd64 on an Athlon.

What do others use?

Thanks,

Doug.

Bookmark/Search this post with:

Average:

best log checker

Submitted by s. keeling on Wed, 2007-01-31 23:00.

Douglas Allan Tutty :
> I'm trying to find a good log checker.
>
> Basically, I want it to report anything that I don't tell it to ignore.

Well, there's always a shell script that looks for date --yesterday
(nonportable), then grep -v 'string1|string2|...' Don't laugh. It's
what I used before logcheck.

> I've tried logcheck first and when I couldn't get it to do what I want I
> tried logwatch. It has an ignore file that it says to just cut and

It does? Mine (sarge/stable) has ignore directories:

drwxr-s--- 2 root logcheck 1024 Oct 23 20:37 ignore.d.paranoid/
drwxr-s--- 2 root logcheck 2048 Aug 12 19:57 ignore.d.server/
drwxr-s--- 2 root logcheck 1024 Aug 12 19:57 ignore.d.workstation/

and the one it uses is defined in logcheck.conf. I was getting really
annoyed at seeing dumb stuff about gconfd, then I noticed I was using
"server" instead of "workstation". The ignore.d.workstation includes
a file "gconf", which lists exactly the junk I don't care about. Doh.

Of course, a server shouldn't be running insecure stuff like X.

> paste what you want to ignore. I do that and it doesn't ignore it.
> Some docs mention that its all based on regular expressions so I tried
> enclosing the lines in quotes to no avial.

Here's a typical useless message (for me):

Oct 9 16:54:42 heretic gconfd (keeling-4010): Resolved address
"xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only
configuration source at
position 0

Here's an entry from gconf:

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ gconfd
$[._[:alnum:]-]+-[0-9]+$: Resolved address "[^[:space:]]+"
to a read-only configuration source at position [^[:space:]]+$

That says:

- at the start of the line ("^")

- three non-whitespace chars ("Oct")

- a space

- the set of space, colon, zero through nine (eleven chars total),
then a space, then the set of period, underscore, alpha-numeric,
or dash/hyphen (more than zero of them "+")

- a space

- the string "gconfd"

- ...

> I _like_ most of what logwatch does, like telling me how many times a
> login happened, especially failed ones. I just don't like to have to
> pour through all the bootup lines every day.

Don't shutdown? Yeah, I know.

--
Any technology distinguishable from magic is insufficiently advanced.
(*) http://www.spots.ab.ca/~keeling Linux Counter #80292
- - http://www.faqs.org/rfcs/rfc1855.html Please, don't Cc: me.
Spammers! http://www.spots.ab.ca/~keeling/emails.html

best log checker

Submitted by Douglas Tutty on Thu, 2007-02-01 00:00.

On Thu, Feb 01, 2007 at 02:55:12AM +0000, s. keeling wrote:
> Douglas Allan Tutty :
> > I'm trying to find a good log checker.
> >
> > Basically, I want it to report anything that I don't tell it to ignore.
>
> Well, there's always a shell script that looks for date --yesterday
> (nonportable), then grep -v 'string1|string2|...' Don't laugh. It's
> what I used before logcheck.
>
> > I've tried logcheck first and when I couldn't get it to do what I want I
> > tried logwatch. It has an ignore file that it says to just cut and
>
> It does? Mine (sarge/stable) has ignore directories:
>
> drwxr-s--- 2 root logcheck 1024 Oct 23 20:37 ignore.d.paranoid/
> drwxr-s--- 2 root logcheck 2048 Aug 12 19:57 ignore.d.server/
> drwxr-s--- 2 root logcheck 1024 Aug 12 19:57 ignore.d.workstation/
>
> and the one it uses is defined in logcheck.conf. I was getting really
> annoyed at seeing dumb stuff about gconfd, then I noticed I was using
> "server" instead of "workstation". The ignore.d.workstation includes
> a file "gconf", which lists exactly the junk I don't care about. Doh.
>
> Of course, a server shouldn't be running insecure stuff like X.
>
> > paste what you want to ignore. I do that and it doesn't ignore it.
> > Some docs mention that its all based on regular expressions so I tried
> > enclosing the lines in quotes to no avial.
>
> Here's a typical useless message (for me):
>
> Oct 9 16:54:42 heretic gconfd (keeling-4010): Resolved address
> "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only
> configuration source at
> position 0
>
> Here's an entry from gconf:
>
> ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ gconfd
> $[._[:alnum:]-]+-[0-9]+$: Resolved address "[^[:space:]]+"
> to a read-only configuration source at position [^[:space:]]+$
>
> That says:
>
> - at the start of the line ("^")
>
> - three non-whitespace chars ("Oct")
>
> - a space
>
> - the set of space, colon, zero through nine (eleven chars total),
> then a space, then the set of period, underscore, alpha-numeric,
> or dash/hyphen (more than zero of them "+")
>
> - a space
>
> - the string "gconfd"
>
> - ...
>
> > I _like_ most of what logwatch does, like telling me how many times a
> > login happened, especially failed ones. I just don't like to have to
> > pour through all the bootup lines every day.
>
> Don't shutdown? Yeah, I know.

Its a workstation. I turn off most of the power at night.

Your exaple is logcheck, which I agree relies on RE, whereas I gave up
on that because of that and tried logwatch which has an ignore file.

I _wish_ that logwatch or logcheck came out-of-the-box able to ignore
ingnorable stuff on a stock debian workstation.

RE has always looked to me like a squirrl has been having lunch on the
keyboard.

Why doesn't someone make a companion interactive rule maker? Run it in
the foreground and have it give you each line it would normally report
and you say yae or nay. From that it could make RE rules.

Doug.

best log checker

Submitted by s. keeling on Thu, 2007-02-01 15:00.

Douglas Allan Tutty :
> On Thu, Feb 01, 2007 at 02:55:12AM +0000, s. keeling wrote:
> > Douglas Allan Tutty :
> >
> > > I've tried logcheck first and when I couldn't get it to do what I want I
> > > tried logwatch. It has an ignore file that it says to just cut and

Ah, crap. Sorry for misreading that.

> Why doesn't someone make a companion interactive rule maker? Run it in

i) This is free software. Go right ahead. :-) Some of the
neatest stuff came into being because someone felt an itch they
couldn't scratch. It's probably a very difficult problem to
solve, though.

ii) Unix-ish OSs have a steep learning curve. The curve pays off
with extraordinary power.

Regular expressions aren't difficult to master. The biggest problem
I've found with them is subsets of them. Shell RE's, perl RE's, awk,
... Some work in all of them, some work in only one of them, some
work differently in each. It can be confusing, but it's really not
that hard.

best log checker

Submitted by Douglas Tutty on Thu, 2007-02-01 17:00.

On Thu, Feb 01, 2007 at 07:15:32PM +0000, s. keeling wrote:
> Douglas Allan Tutty :
> > On Thu, Feb 01, 2007 at 02:55:12AM +0000, s. keeling wrote:
> > > Douglas Allan Tutty :
> > >
> > Why doesn't someone make a companion interactive rule maker? Run it in
>
> i) This is free software. Go right ahead. :-) Some of the
> neatest stuff came into being because someone felt an itch they
> couldn't scratch. It's probably a very difficult problem to
> solve, though.
>
> ii) Unix-ish OSs have a steep learning curve. The curve pays off
> with extraordinary power.
>
> Regular expressions aren't difficult to master. The biggest problem
> I've found with them is subsets of them. Shell RE's, perl RE's, awk,
> ... Some work in all of them, some work in only one of them, some
> work differently in each. It can be confusing, but it's really not
> that hard.
>

However,

Even on my 486, my brute-force log checker completes in under a minute.
It may be worth it if RE would save an hour or so.

I still have trouble with conditional stuff (like if) in bash. I use
bash scripting like dos .bat files. If (so to speak) I need
conditionals, I switch to python. I don't like languages where having
two spaces instead of one (or none if beside a bracket) creates an
error. Having a whole line consist of mostly punctuation (like your RE
example) makes me think my printer is on the fritz.

Doug.

best log checker

Submitted by Roberto C. Sanchez on Wed, 2007-01-31 23:00.

On Wed, Jan 31, 2007 at 08:47:20PM -0500, Douglas Allan Tutty wrote:
>
> I do neither perl nor RE: they're both too cryptic. I guess I'll never
> be a true *N*X weenie.
>
So, you want to be able to parse logs and yet don't want to learn the
most powerful regex syntax for that? Do you know that the original
purpose of Perl was log file parsing? If I were you, I'd invest in an
O'Reilly book and spend a few days learning Perl RE syntax.

Regards,

-Roberto

--
Roberto C. Sanchez
http://people.connexer.com/~roberto
http://www.connexer.com

best log checker

Submitted by Douglas Tutty on Thu, 2007-02-01 00:00.

On Wed, Jan 31, 2007 at 10:21:37PM -0500, Roberto C. Sanchez wrote:
> On Wed, Jan 31, 2007 at 08:47:20PM -0500, Douglas Allan Tutty wrote:
> >
> > I do neither perl nor RE: they're both too cryptic. I guess I'll never
> > be a true *N*X weenie.
> >
> So, you want to be able to parse logs and yet don't want to learn the
> most powerful regex syntax for that? Do you know that the original
> purpose of Perl was log file parsing? If I were you, I'd invest in an
> O'Reilly book and spend a few days learning Perl RE syntax.
>

My python script took an ignore file, lines of strings to ignore.

It took the log file and for each line in the log file, checked for the
ignore strings. If it didn't find any it included the log line in the
report.

Like I said, brute force. Hardly elegant but it worked.

I could use RE in python but its still goblygook. I've tried on several
occasions to learn both RE and perl with no lasting results. Ditt C.
Give me python and Fortran77.

Doug.

best log checker

Submitted by Chris Bannister on Sun, 2007-02-04 07:00.

On Wed, Jan 31, 2007 at 10:21:37PM -0500, Roberto C. Sanchez wrote:
> most powerful regex syntax for that? Do you know that the original
> purpose of Perl was log file parsing? If I were you, I'd invest in an

I thought Perl was written to handle Usenet news posts.

--
Chris.
======
Don't forget to check that your /etc/apt/sources.lst entries point to
etch and not testing, otherwise you may end up with a broken system once
etch goes stable.

best log checker

Submitted by Paul Johnson on Thu, 2007-02-01 22:00.

Douglas Allan Tutty wrote:

> I'm trying to find a good log checker.
>
> Basically, I want it to report anything that I don't tell it to ignore.
>
> I've tried logcheck first and when I couldn't get it to do what I want I
> tried logwatch. It has an ignore file that it says to just cut and
> paste what you want to ignore. I do that and it doesn't ignore it.
> Some docs mention that its all based on regular expressions so I tried
> enclosing the lines in quotes to no avial.
>
> I do neither perl nor RE: they're both too cryptic. I guess I'll never
> be a true *N*X weenie.

regexp's are much easier to learn quickly with kregexpeditor.

best log checker

Submitted by Sven Arvidsson on Fri, 2007-02-02 13:00.

On Wed, 2007-01-31 at 20:47 -0500, Douglas Allan Tutty wrote:
> I'm trying to find a good log checker.
>
> Basically, I want it to report anything that I don't tell it to ignore.
>
> I've tried logcheck first and when I couldn't get it to do what I want I
> tried logwatch. It has an ignore file that it says to just cut and
> paste what you want to ignore. I do that and it doesn't ignore it.
> Some docs mention that its all based on regular expressions so I tried
> enclosing the lines in quotes to no avial.
>
> I do neither perl nor RE: they're both too cryptic. I guess I'll never
> be a true *N*X weenie.
[...]
> What do others use?

But... regular expressions can be so important!
http://xkcd.com/c208.html

Humor aside, I seem to forget most of the regular expressions I learn
very quickly. Even so, I use logcheck and find it to be quite good.

It's not so hard to write general rules that work on my system (but
would probably be too general for anyone else) simply by reading and
adapting the existing rules.

Also make sure you check out the README in logcheck-database, especially
the section about testing rules as it gives you a handy one liner for
trying out regex, making sure they match.

--
Cheers,
Sven Arvidsson
http://www.whiz.se
PGP Key ID 760BDD22

Navigation

User login

Spam?

best log checker

Comment viewing options

best log checker

best log checker

best log checker

best log checker

best log checker

best log checker

best log checker

best log checker

best log checker

Useful Debian Links