Massive log files crashed linux

MartinT's picture

Forums: 

I just installed Debian 8. I set it to download all security updates and everything was OK for a few days. The only non Debian software I installed was Flash. Then I ran out of memory and most programs stopped working properly. I had to switch the machine off and now it won't reboot.

Using a Knoppix rescue disk I found that wpa_supplicant seems to be the problem:

$ cd /media/sda6/var/log;
$ ls -la -S | head -12
total 759485332
-rw-r----- 1 root adm 221536518950 Jan 3 05:12 syslog
-rw-r----- 1 root adm 101853816379 Dec 31 17:52 kern.log.1
-rw-r----- 1 root adm 90624523220 Dec 31 17:52 daemon.log.1
-rw-r----- 1 root adm 83930297342 Jan 3 05:12 kern.log
-rw-r----- 1 root adm 76900787962 Dec 31 17:52 messages.1
-rw-r----- 1 root adm 74319636872 Jan 3 05:12 daemon.log
-rw-r----- 1 root adm 63366348453 Jan 3 05:12 messages
-rw-r----- 1 root adm 62698422582 Dec 23 07:58 syslog.1
-rw-r----- 1 root adm 2084335688 Dec 22 07:35 syslog.2.gz
-rw-r----- 1 root adm 394985472 Dec 24 07:40 syslog.1.gz
-rw-r--r-- 1 root root 1098000 Dec 31 18:16 dpkg.log.1

$ tail -8 syslog; echo
Jan 3 06:45:35 Local wpa_supplicant[933]: Could not set interface wlan0 flags (UP): Cannot allocate memory
Jan 3 06:45:35 Local wpa_supplicant[933]: nl80211: Could not set interface 'wlan0' UP
Jan 3 06:45:35 Local wpa_supplicant[933]: Could not set interface wlan0 flags (UP): Cannot allocate memory
Jan 3 06:45:35 Local wpa_supplicant[933]: WEXT: Could not set interface 'wlan0' UP
Jan 3 06:45:35 Local wpa_supplicant[933]: wlan0: Failed to initialize driver interface
Jan 3 06:45:35 Local NetworkManager[607]: [1451803535.856771] [supplicant-manager/nm-supplicant-interface.c:856] interface_add_cb(): (wlan0): error adding interface: wpa_supplicant couldn't grab this interface.
Jan 3 06:45:35 Local NetworkManager[607]: (wlan0): supplicant interface state: starting -> down
Jan 3 06:45:35 Local wpa_supplicant[933]: Could n

$ tail -100000000 syslog | grep 'wpa_supplicant.*Cannot allocate memory' | wc -l
13367638
$ head -100000000 syslog | grep 'wpa_supplicant.*Cannot allocate memory' | wc -l
13333176

I am not sure where to report the bug; or what to do to stop this happening again if I reinstall Debian.

Logs filling up things...

IntnsRed's picture

From what you've provided, it seems there's a bug or problem in the wireless networking that is causing the "wpasupplicant" package to spew out log entries which are filling your drive.

The first thing I'd do is to remove some of those large log files. You can do that as root and/or from the Knoppix rescue disk. You'll find them in /var/log, of course. (You can also delete the archived *.gz and/or the *.1 log files too.) Freeing up that disk space will likely result in the system booting again.

Are you using wireless networking? If so, did the wireless networking work properly for those few days when the computer worked fine? If it did work, did anything on the networking side of things change between when it was working and when it started having problems?

If you're not using wireless networking, simply removing/purging the "wpasupplicant" package would likely stop it from spamming the logs. But of course, that is ignoring whatever is the real issue.

You can find the Debian bug system here, and the wpasupplicant package page here.

Thank You!

MartinT's picture

I deleted the largest log files and could could boot again. wpasupplicant was still generating lots of messages, but after the next reboot it stopped. However I do not use wireless netwoking, so after a bit of research I removed the package.

Logs filling up things...

IntnsRed's picture

That's good that the problem is solved.

But still, I'd suggest filing a bug report on the wpasupplicant package. Letting the package maintainer know about the problem and what hardware it happened will hopefully cause the issue to be fixed and may be info the maintainer is seeking.

Finally sorted

MartinT's picture

The day after I removed wpasupplicant I lost all internet access so I reinstalled it. This brought back the internet, but the log files were soon back and growing. Eventually I discovered that the network-management widget which I had added to the KDE desktop enabled wireless networking by default. I disabled it and the log files stopped growing.
Before I got round to posting a bug report a new version of wpasupplicant was released; the bug has been fixed.

On the plus side, I now understand a few things better than I did before; and I know how the bug-reporting works for future reference Smile