[Vserver] Re: Chasing kernel crashes on SMP

From: Grzegorz Nosek <grzegorz.nosek_at_gmail.com>
Date: Mon 28 Nov 2005 - 11:02:51 GMT
Message-ID: <121a28810511280302p115dc1b5l@mail.gmail.com>

2005/11/28, Grzegorz Nosek <grzegorz.nosek@gmail.com>:
> Hello
>
> I'd like to report on my findings in my continuing crusade to find the
> cause of AMD64 kernel crashes.
>
> First, it still crashes.
>
> Second, but now I have an oops trace :)
>
> Third, it's not AMD64-specific after all (though it seems much more
> frequent there)
>
> Fourth, since the last-but-one build (internal rev17) the oopses seem
> more frequent.
>
> As I've booted my test box (dual Xeon) with rev17, it found two extra
> CPUs (I enabled ACPI in rev14 and it was running rev13 before) and
> started crashing quite frequently (sometimes reaching uptime of only a
> few minutes). I'm running the box with rev13 now (no ACPI, sees 2 CPUs
> only) and it's at least usable (though it probably *will* crash sooner
> or later :))
>
> The crash occurs in fs/proc/array.c:do_task_stat(), triggered by
> pidof. It is clearly a NULL pointer dereference. I have attached an
> oops from the amd64smp.17 kernel and a dump of do_task_stat assembly
> code from amd64smp.18 (these two builds only differ in Fusion MPT SCSI
> support so this file should be identical) with the oopsing instruction
> marked.
>
> The p4 kernel crashes in the very same assembly instruction.
>
> I'm off to relate the assembly to the kernel source. I'll report as
> soon as I find something but I wanted to share this with vserver-gurus
> (it'll probably be easier to spot the mistake for you).
>
> Best regards,
> Grzegorz Nosek
>
> PS. Sorry for the largish attachments :)
>
>
>

Hello

It seems that my original message is still awaiting moderator approval
due to attachment size but I moved a bit forward with tracing the
oops. Apparently do_task_state gets passed a null task_struct pointer
as the first argument and crashes upon entering get_task_state (which,
incidentally, is the first real instruction in this function).

It looks like a race condition where the task_struct is freed "just
before" the call to proc_t[g]id_stat and do_task_stat.

Does the following look like an acceptable fix for this issue? I'll
throw it into a test kernel and give it a try, but I'd really
appreciate your comments.

Best regards,
Grzegorz Nosek

--- fs/proc/base.c.orig 2005-11-25 00:07:43.000000000 +0100
+++ fs/proc/base.c 2005-11-28 11:44:11.000000000 +0100
@@ -718,6 +718,9 @@
        ssize_t length;
        struct task_struct *task = proc_task(inode);

+ if (!task)
+ return -ENOENT;
+
        if (count > PROC_BLOCK_SIZE)
                count = PROC_BLOCK_SIZE;
        if (!(page = __get_free_page(GFP_KERNEL)))
_______________________________________________
Vserver mailing list
Vserver@list.linux-vserver.org
http://list.linux-vserver.org/mailman/listinfo/vserver
Received on Mon Nov 28 11:03:17 2005

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Mon 28 Nov 2005 - 11:03:46 GMT by hypermail 2.1.8